Updated YASS container

[alejoe91]

Updates

• Updated and tested YASS docker
• Added build.sh and push.sh
• Fixes entrypoint for conda environment

Notes

• this commit is checked out because of an issue with master
• In order to run the container, the command needs to be: docker run --gpus all -it --entrypoint /bin/bash yass:2.0.0
  • the --gpus all ensures the GPU(s) are used correctly in the container
  • --entrypoint /bin/bash is needed to run the entrypoint script, which activates the right environment

@yger can you test this? @samuelgarcia for SI docker runs, I suggest we add a class variables docker_requires_gpu and docker_requires_entrypoint flags to modify the docker commands. Should be straightforward (https://docker-py.readthedocs.io/en/stable/containers.html)

[yger]

Ok, will do that to test asap

[yger]

I've installed the docker, will wait for Sam to make the adjsutement in the run_sorter command because otherwise, as far as i understood, you are telling me that such a command will not work, will it? Will have a look meanwhile. Once this is working, I can relaunch evertyhing and have all sorters for the paper

[alejoe91]

@yger I can also give it a try on monday, but let's wait for Sam and ask him if he likes the approach. You know he can be difficult :P

[yger]

Currently, with a naive run (and the new docker built, with only 7.5Go instead of 12Go), I still have errors

docker.errors.ContainerError: Command 'sh -c "pip install --upgrade --force MEArec && pip install --upgrade --force https://github.com/samuelgarcia/spikeinterface/archive/big_refactoring.zip && pip install --upgrade --force https://github.com/NeuralEnsemble/python-neo/archive/master.zip && python /sorting_output_folder/in_docker_sorter_script.py && chown 1001:1001 -R /sorting_output_folder/sorting_yass_docker"' in image 'spikeinterface/yass:latest' returned non-zero exit status 1: b'Traceback (most recent call last):\n File "/sorting_output_folder/in_docker_sorter_script.py", line 17, in \n verbose=True, raise_error=True, **sorter_params)\n File "/home/si/miniconda3/lib/python3.7/site-packages/spikeinterface/sorters/runsorter.py", line 73, in run_sorter_local\n output_folder = SorterClass.initialize_folder(recording, output_folder, verbose, remove_existing_folder)\n File "/home/si/miniconda3/lib/python3.7/site-packages/spikeinterface/sorters/basesorter.py", line 110, in initialize_folder\n output_folder.mkdir(parents=True, exist_ok=True)\n File "/home/si/miniconda3/lib/python3.7/pathlib.py", line 1268, in mkdir\n self._accessor.mkdir(self, mode)\nPermissionError: [Errno 13] Permission denied: '/sorting_output_folder/sorting_yass_docker'\n'

[yger]

But I guess this might be because I did not pass the extra argument you mentioned. I'll have a look today at the core of spikeinterface to see how I can pass them

[alejoe91]

@yger can you try this SI PR? SpikeInterface/spikeinterface#195

[yger]

My bad. Now I have this error with your PR

Traceback (most recent call last):
File "/home/cure/.local/lib/python3.6/site-packages/docker/api/client.py", line 268, in _raise_for_status
response.raise_for_status()
File "/home/cure/.local/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localhost/v1.41/containers/2276c03ef662bb49125dbe65d81aed8c9f185c02df46a5dc73676d1ba964ce56/start

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "test_yass.py", line 10, in
verbose=True, raise_error=True, docker_image=docker_image)
File "/media/cure/Secondary/pierre/softwares/spikeinterface/spikeinterface/sorters/runsorter.py", line 61, in run_sorter
verbose=verbose, raise_error=raise_error, **sorter_params)
File "/media/cure/Secondary/pierre/softwares/spikeinterface/spikeinterface/sorters/runsorter.py", line 221, in run_sorter_docker
res = client.containers.run(docker_image, command=command, volumes=volumes, **extra_kwargs)
File "/home/cure/.local/lib/python3.6/site-packages/docker/models/containers.py", line 818, in run
container.start()
File "/home/cure/.local/lib/python3.6/site-packages/docker/models/containers.py", line 404, in start
return self.client.api.start(self.id, **kwargs)
File "/home/cure/.local/lib/python3.6/site-packages/docker/utils/decorators.py", line 19, in wrapped
return f(self, resource_id, *args, **kwargs)
File "/home/cure/.local/lib/python3.6/site-packages/docker/api/container.py", line 1111, in start
self._raise_for_status(res)
File "/home/cure/.local/lib/python3.6/site-packages/docker/api/client.py", line 270, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "/home/cure/.local/lib/python3.6/site-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error for http+docker://localhost/v1.41/containers/2276c03ef662bb49125dbe65d81aed8c9f185c02df46a5dc73676d1ba964ce56/start: Internal Server Error ("could not select device driver "" with capabilities: [[gpu]]")

Will try to update the nvidia drivers and see if this is the problem

[yger]

answer is nope. I have a GPU, recognized by nvidia-smi command, but don't know how to pass it to the docker image obvisouly...

[alejoe91]

@yger if you run this docker run --gpus all -it --entrypoint /bin/bash spikeinterface/yass:2.0.0 can you access the GPU with nvidia-smi? It works on my side!

[yger]

Im' not a docker expert.... But if I do
docker run --gpus all -it --entrypoint /bin/bash spikeinterface/yass

I got the same error immediatly

docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].

Maybe I should rebuilt everthing?

[yger]

My bad, I found the solution I think, it was related to the docker install, and not to spikeinterface. Now it seems to work. Next step is then simply to have a run_sorter command that will take into account the docker images. Or is it already the case? I'll check

[yger]

And of course, as usual, I spoke too soon. This is the next error, that happened after a while, so docker image was properly launched

docker.errors.ContainerError: Command 'sh -c "pip install --upgrade --force MEArec && pip install --upgrade --force https://github.com/SpikeInterface/spikeinterface/archive/master.zip && pip install --upgrade --force https://github.com/NeuralEnsemble/python-neo/archive/master.zip && python /sorting_output_folder/in_docker_sorter_script.py && chown 1001:1001 -R /sorting_output_folder/sorting_yass_docker"' in image 'spikeinterface/yass:latest' returned non-zero exit status 1: b'Traceback (most recent call last):\n File "/sorting_output_folder/in_docker_sorter_script.py", line 17, in \n verbose=True, raise_error=True, **sorter_params)\n File "/home/si/miniconda3/lib/python3.7/site-packages/spikeinterface/sorters/runsorter.py", line 74, in run_sorter_local\n output_folder = SorterClass.initialize_folder(recording, output_folder, verbose, remove_existing_folder)\n File "/home/si/miniconda3/lib/python3.7/site-packages/spikeinterface/sorters/basesorter.py", line 111, in initialize_folder\n output_folder.mkdir(parents=True, exist_ok=True)\n File "/home/si/miniconda3/lib/python3.7/pathlib.py", line 1268, in mkdir\n self._accessor.mkdir(self, mode)\nPermissionError: [Errno 13] Permission denied: '/sorting_output_folder/sorting_yass_docker'\n'

[alejoe91]

Yes this is the same erro I got...I think that the problem is that I install miniconda and related packages from a non-root user: https://github.com/SpikeInterface/spikeinterface-dockerfiles/blob/yass_update/yass/Dockerfile#L18-L38

I don't have time today but next week I'll try to install everything directly in root. Or if you want to play around with it Pierre you can give it a try! Let me know :)

[alejoe91]

@yger @samuelgarcia after the last commit the YASS docker seems to work! :)

You can just build it from the folder (sh build.sh) and then run it directly via SI (you need this PR as there was a bug in the yassextractor):

sort_yass = si.run_yass(recording, docker_image="spikeinterface/yass-base:2.0.0")

[yger]

Great ! Looking forward to try that next week. Sam, can you merge Alessio's PR into my branch?

[samuelgarcia]

@yger : this is dolcker files here only here.

@alejoe91 : Can i merge this now ?