-
Notifications
You must be signed in to change notification settings - Fork 81
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
git-lfs missing from container #14
Comments
Hi @KatharinaHoff, thanks so much! I forgot this dependency, which I believe is just for loading the pretrained weights from Huggingface. Good catch :) I've made the change and uploaded a new Docker image (reflected in the readme). You can now pull with (I removed the 'public' name):
Enjoy! |
Dear @exnx , thank you so much for trying to fix it. Sadly, the pip install of git-lfs did not fix the problem. Sorry, my bad, I had not tested my first suggestion. I have now figured out how to fix it. Possibly not the most elegant way... but if you append the following to your Dockerfile, then both the docker and the singularity built contain git lfs and models can be loaded from huggingface:
This solution, I have tested both with Docker and Singularity. git lfs works. Maybe you also want to add the Singularity instructions (adapt from my initial post here) to the Readme.md? Just an idea to save other people some time. I tested it with singularity-ce version 3.11.3 , all works well. |
Thanks for the update! I haven't been able to test it out myself, but I'll report back when I do. |
I ended making a second Docker image with the Nucleotide Transformer datasets, and weights to reproduce the results from our paper. This new image includes the correct git-lfs dependency for pulling in weights from Huggingface. You can find the image here:
To build the image, I used tips from this thread, which basically just means adding:
Eventually I'll add this to the main Dockerfile in the repo, but for now there are 2 Docker images. |
Hi @KatharinaHoff and @exnx , thanks a lot for posting directions on how to generate and use a Singularity image of HyenaDNA! I tried as Katharina suggested:
But getting |
The steps above by Katharina didn't work for me. Instead I used a different set of commands, which you can find in this image instead, on dockerhub. You can find the steps in the readme. I'm not especially familiar with Singularity, but there's a different step when starting the image that tunnels your local directory to the container, otherwise you're just getting the environment, but not any of the code.
|
Great, thank you!
As first thing, I tried the nt7 version also available - but the converted
sif image was giving errors. Will try converting nt6 and let you know.
…On Tue, Nov 14, 2023, 9:03 PM Eric Nguyen ***@***.***> wrote:
The steps above by Katharina didn't work for me. Instead I used a
different set of commands, which you can find in this image instead.
hyenadna/hyena-dna-nt6:latest
—
Reply to this email directly, view it on GitHub
<#14 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB7IT6G3WJBOD6DH6K6FYKTYEREJFAVCNFSM6AAAAAA4GHLP6SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJRHAYTGNRUGI>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
The But specifically, the commands you want in the Dockerfile are:
I updated the Dockerfile in this repo, it now includes this command if you build your own image. |
Thanks! I re-run with
This strange 'not a valid test operator' is the same I was getting with the sif image of Wish I could use Docker - but I am stuck with Singularity on the HPC A100 nodes I have available. |
ChatGPT? ie, check for how to make the docker cmd provided into an equivalent singularity cmd. Sorry, we don't support singularity on our end, we just don't use it. |
eg
|
No worries, thank you for all the help. I think I found the culprit - my cloned hyenaDNA folder was on a mounted drive (/mnt/ etc.) that for some reason wasn't accessible by the container image. Now moved everything on my home folder and it seems to work. |
Hi!
Thanks for this very cool repository, the preprint is very cool, too!
I am a total beginner to using relevant machine learning libraries, but I think the git-lfs is missing from the docker container of hyena-dna. Admittedly, I did a bit of weird stuff with it because I can't execute docker on the HPC. I converted it to Singularity. Nevertheless, I'd expect to be able to call git-lfs from there, too, and it seems to be missing.
Here's what I did:
Error:
I think it could be fixed by adding git-lfs to the requirements.txt, and then rebuilding the container.
The text was updated successfully, but these errors were encountered: