TFHub Deployment

[Mik-TF]

Todo

• Develop procedures and docs to
  • deploy TFHub (lets people go from docker to tfgrid ZOS flists)

Related issue

• See tfgrid validator stack main project issue for overview

[coesensbert]

in progress: https://github.com/threefoldtech/tf_operations/issues/2113

what is meant with "lets people go from docker to tfgrid ZOS flists" ?

[Mik-TF]

It means people can turn Docker into Flist.

We have this by default on the hub, as I understand, here: https://hub.grid.tf/docker-convert.

So it's integrated in the hub. Nothing "new"/more to do in that sense.

[coesensbert]

It means people can turn Docker into Flist.

We have this by default on the hub, as I understand, here: https://hub.grid.tf/docker-convert.

So it's integrated in the hub. Nothing "new"/more to do in that sense.

ok good, this was enabled here https://github.com/threefoldtech/grid_deployment/blob/development/grid-hub/docker-compose.yml#L43

[coesensbert]

this last commit has a chance to work: 90136c8

Need to go over a few things with @maxux and needs to be tested. Since all data is already on the second hub server (https://02.hub.grid.tf/) and it's not in use yet, I will take it down and apply changed for our compose stack to work there. So we have our second hub in a 'complaint' setup.

[coesensbert]

stuck at

root@horizon-old:~/code-hub/grid_deployment/grid-hub# docker compose --env-file .env up -d
invalid interpolation format for services.0-db.image.
You may need to escape any $ with another $.
ghcr.io/threefoldtech/0-db:${0-DB_IMG}

[coesensbert]

I'm trying out some installs and tricks to get around a few open issues.
There also seems to be some confusion on how all of this should be build and operate. Will wait on further feedback to continue

[Mik-TF]

Thanks @coesensbert for the update. Let's talk this week on TG so we can see how to move forward.

We need to make everything clear and check how to proceed further.

Ideally we can finish the project by the end of July: https://github.com/orgs/threefoldtech/projects/211/views/1

I'll try to help you on this in the coming weeks, to the best of my knowledge. Thanks again!

[coesensbert]

stuck at

root@horizon-old:~/code-hub/grid_deployment/grid-hub# docker compose --env-file .env up -d
invalid interpolation format for services.0-db.image.
You may need to escape any $ with another $.
ghcr.io/threefoldtech/0-db:${0-DB_IMG}

resolved by fae798b . Seems one can not use numbers for env vars.

[coesensbert]

while trying to deploy I'm stuck at this. Is this path required @maxux ? So should be defined as volume in the compose file? https://github.com/threefoldtech/grid_deployment/blob/development/grid-hub/docker-compose.yml#L39-L43

[+] userdata directory: /public
[+] workdir directory : /workdir
[+] upload directory  : /workdir/distfiles
[+] flist creation    : /workdir/temp
[+] docker creation   : /workdir/temp
[+] public directory  : /public/users
[+] zflist binary path: /opt/0-flist/zflist/zflist
[+] flist: running 0-flist self-check
['/opt/0-flist/zflist/zflist', 'init']
Traceback (most recent call last):
  File "/hub/src/flist-uploader.py", line 85, in <module>
    if not hc.check():
  File "/hub/src/hub/flist.py", line 334, in check
    test = self.execute("init")
  File "/hub/src/hub/flist.py", line 78, in execute
    p = subprocess.Popen(command, env=self.environ, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  File "/usr/lib/python3.10/subprocess.py", line 971, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.10/subprocess.py", line 1863, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/opt/0-flist/zflist/zflist'

[Mik-TF]

Please in time lmk the status of this issue.

Note: the TFHub should be for a micro VM (not full VM), as the micro VM is the basis for flist and dashboard apps.

Once this is done, I will turn the procedure/docker compose into an flist and then dashboard app (see here: threefoldtech/home#1563)

[maxux]

About zflist-bin: https://github.com/threefoldtech/0-hub/blob/master/deployment/README.md?plain=1#L9-L10
Just set it to /usr/bin/zflist on your config file and it should works :)

[coesensbert]

About zflist-bin: https://github.com/threefoldtech/0-hub/blob/master/deployment/README.md?plain=1#L9-L10 Just set it to /usr/bin/zflist on your config file and it should works :)

great thx, that works! Next one from the hub container

[+] userdata directory: /public
[+] workdir directory : /workdir
[+] upload directory  : /workdir/distfiles
[+] flist creation    : /workdir/temp
Traceback (most recent call last):
  File "/hub/src/flist-uploader.py", line 107, in <module>
[+] docker creation   : /workdir/temp
[+] public directory  : /public/users
[+] zflist binary path: /usr/bin/zflist
[+] flist: running 0-flist self-check
['/usr/bin/zflist', 'init']
Code: 0, b'{"success": true, "error": null, "response": {}}'
Error: b''
['/usr/bin/zflist', 'metadata', 'backend', '--host', '0-db', '--port', '9900', '--password', '', '--namespace', 'default']
Code: 0, b'{"success": true, "error": null, "response": {}}'
Error: b''
['/usr/bin/zflist', 'check']
Code: 0, b'{"success": true, "error": null, "response": {"content": [], "regular": 0, "symlink": 0, "directory": 0, "special": 0, "failure": 0, "fullsize": 0}}'
Error: b''
    hub.threebot.configure(app,
  File "/hub/src/hub/threebot.py", line 194, in configure
    auth = ThreeBotAuthenticator(app, appid, privatekey, signseed, vusers)
  File "/hub/src/hub/threebot.py", line 34, in __init__
    self.privkey = nacl.public.PrivateKey(privatekey, nacl.encoding.Base64Encoder)
  File "/usr/lib/python3/dist-packages/nacl/public.py", line 98, in __init__
    raise exc.TypeError(
nacl.exceptions.TypeError: PrivateKey must be created from a 32 bytes long raw secret key

[maxux]

Generate threebot-privatekey:

openssl genpkey -algorithm x25519 -out private.key
openssl pkey -in private.key -text | xargs | sed -e 's/.*priv\:\(.*\)pub\:.*/\1/' | xxd -r -p | base64

To generate the seed threebot-seed:

python -c "import nacl; from nacl import utils; print(nacl.utils.random(32))"

[coesensbert]

we resolved most issues, two remain:

• all username flist links seem to work. When navigating one has to login and is redirected to https://0-hub:5555/login-method . Not sure where it comes from. tf connect should be the default. When navigating to 'login' > threefold connect that seems to work
  compared config multiple times with https://02.hub.grid.tf/ (first/manual backup hub for now) but found no differences. Env var $DOMAIN also seems to work as intended

• used latest incremental.py from https://github.com/threefoldtech/0-db/blob/development-v2/tools/incremental-update/incremental.py but this does not work. Adjusted it accordingly here https://github.com/threefoldtech/grid_deployment/blob/development/grid-hub/incremental.py#L108-L110
  0-db is exposed to 127.0.0.1:9900
  when running:

python3 incremental.py
root@bt-dc-test3:~/code/grid_deployment/grid-hub# python3 incremental.py
Traceback (most recent call last):
  File "/root/code/grid_deployment/grid-hub/incremental.py", line 108, in <module>
    incremental = ZDBIncremental("hub.grid.tf", 9900, "127.0.0.1", 9900)
  File "/root/code/grid_deployment/grid-hub/incremental.py", line 19, in __init__
    target.set_response_callback("NSINFO", redis._parsers.helpers.parse_info)
AttributeError: module 'redis' has no attribute '_parsers'

you can test by:

• pointing a domain to your pub ip and set it in .env
• open port 80, 443 and 9900
• clone repo and sh install-hub.sh

a test instance with hub and bootstrap:
https://hub.ninja.tf/
https://bootstrap.ninja.tf/

[coesensbert]

all issues resolved, we got a working release: https://github.com/threefoldtech/grid_deployment/tree/development/grid-hub

Need to write docs next

[Mik-TF]

You rock. Good job.

LMK if you need help for the docs.

[coesensbert]

first docs: https://github.com/threefoldtech/grid_deployment/tree/development/grid-hub
can be tested now

[Mik-TF]

Great. Will test early next week!

[coesensbert]

all docs finished: https://github.com/threefoldtech/grid_deployment/blob/development/grid-hub/readme.md
Thx @maxux for all the help!

deploying two new read-only instances: https://github.com/threefoldtech/tf_operations/issues/2733