dvc init in MatrixDS environment

[dmpetrov]

dvc init got stuck in MatrixDS environment:

One of guesses - it is related to NFS:

That's all I know for today :(
@FavioVazquez might have more details.

[FavioVazquez]

Thanks @dmpetrov. Tagging @isaacfab here too

[isaacfab]

That's a good guess about issues related to the NFS. We have had to tweak the user privileges for many of the containers.

[efiop]

Hi @isaacfab ! For how long did you leave it running?

[FavioVazquez]

It ran for a long time, but nothing happened. I couldn't even use Ctrl+C @efiop

[efiop]

@FavioVazquez Right, if you were not able to CTRL+C, then the program was still in kernel space, meaning that it is probably an NFS issue. Is it still running? If not, mind running it again and then:

1. find it's PID with something like ps xaf | grep dvc;
2. show us cat /proc/$PID/stack;

Thanks,
Ruslan

[shcheklein]

@efiop :

matrix@tool-5ca6d6e08f48fd83c6cc6069-mcn6f:/home/test$  cat /proc/248/stack
cat: /proc/248/stack: Operation not permitted

trace is disabled in docker by default.

@isaacfab is there a way to run one of your docker images with NFS with SYS_PTRACE enabled? We need to see the stack trace at least to understand what's happening. Otherwise it's extremely hard to debug this.

I also've tried to run NFS on my machine (server and client), dvc worked fine.

[efiop]

@shcheklein Is that docker image running on linux host?

[shcheklein]

@efiop most likely, yes. @isaacfab do you have more details?

[isaacfab]

The container is running on GCP GKE so yes. All of our containers (with docker files) can be found here. https://github.com/matrixds/tools

[isaacfab]

You can use custom containers on the site (see the repo for instructions). Feel free to enable whatever variables you think would be useful. Have you checked the logs on the environments from the webui?

[shcheklein]

@isaacfab I've checked them, don't see anything interesting.

Re custom tool - I'm not sure I understand which field should I be using to pass docker run arguments. Is there any documentation how those options translate to CLI and how could I adjust for the docker run?

[shcheklein]

@isaacfab to be more specific, we need something like docker run --cap-add SYS_PTRACE ...

[isaacfab]

in the UI for the custom tool, you can add a specific docker run command. here is an example.

[isaacfab]

you could also build the container from scratch and change the run command in the entrypoint script. If you put a test container on docker hub you can build it from the custom UI. (see the example above I've just referenced our current jupyterlab image.