Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce docker image size for the NN grobid version #1088

Merged
merged 2 commits into from Mar 11, 2024
Merged

Conversation

lfoppiano
Copy link
Collaborator

@lfoppiano lfoppiano commented Feb 28, 2024

This PR attempts to reduce the docker image by reducing the number of layers.

By just chancing the layers, I managed to save 700Mb. We could optimise more in future, especially considering that grobid modules (e.g. grobid-quantities) are not able to build properly at the moment, on github actions (limited to ~20Gb).

One solution would be to move everything in the build and then just copy the resulting files in the tensorflow image.

I also noticed that the delft used here is 0.3.3, I'm not sure whether it's by design, I propose to update it to 0.3.4.

@kermitt2 kermitt2 merged commit cb10576 into master Mar 11, 2024
8 checks passed
@kermitt2
Copy link
Owner

Thank you very much Luca, this was indeed requiring some improvements !
The changes in DeLFT 0.3.4 did not impact Grobid, but good to update anyway.

@lfoppiano lfoppiano deleted the reduce-image-size branch March 12, 2024 00:57
@lfoppiano
Copy link
Collaborator Author

@kermitt2 FYI I've created a new temporary image: lfoppiano/grobid:0.8.0-full-slim that I need to be able to build grobid-quantities through github actions.

As a reference the effficiency climbed from 88% to 91% according to https://github.com/wagoodman/dive

@lfoppiano
Copy link
Collaborator Author

Two more noted:

  1. delft 0.3.4 is in pipy but not tagged in the repository
  2. it seems that blingfire has been added in the reader.py, however is not specified in the requirements.txt so any grobid docker image are failing when running using delft

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants