Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically download embeddings #79

Closed
kermitt2 opened this issue Dec 28, 2019 · 1 comment · Fixed by #120
Closed

Automatically download embeddings #79

kermitt2 opened this issue Dec 28, 2019 · 1 comment · Fixed by #120

Comments

@kermitt2
Copy link
Owner

kermitt2 commented Dec 28, 2019

As part of the embeddings and model management in DeLFT, download on demand word embeddings, contextualized embeddings and transformer models.

Thanks to @de-code, the functionality to automatically download embeddings is already available !

@de-code
Copy link
Contributor

de-code commented Jan 3, 2020

Just to add that to that, I've also added preloading of embedding as part of the Docker entrypoint via an environment variable (PRELOAD_EMBEDDING).

I did try it with xz (LZMA) compressed files but found it took too long to decompress. I then reverted to using the decompressed mdb (LMDB cache) even though it was more than twice the size. But that probably depends on the setup and network connection.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants