Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Build more efficient tensorflow-notebook images #472

Closed
ankushagarwal opened this issue Mar 21, 2018 · 1 comment
Closed

Build more efficient tensorflow-notebook images #472

ankushagarwal opened this issue Mar 21, 2018 · 1 comment
Labels
area/jupyter Issues related to Jupyter help wanted priority/p3 Very low priority.

Comments

@ankushagarwal
Copy link
Contributor

The current cpu images are ~10GB and the current gpu images are ~14GB.

We should try to build smaller images.

Here is the Dockerfile used to build these images : https://github.com/kubeflow/kubeflow/blob/master/components/tensorflow-notebook-image/Dockerfile
@ankushagarwal ankushagarwal added help wanted area/jupyter Issues related to Jupyter priority/p3 Very low priority. labels Mar 21, 2018
@jlewi
Copy link
Contributor

jlewi commented Mar 27, 2018

Its possible we have a bunch of intermediary layers as a result of the build process that are storing files that were subsequently deleted in later layers.

I think a quick check is you can use docker to export the image to a tarball and then reimport it. I believe that flattens all the layers.

You can then look at the size of the reimported image and see if its smaller.

@ankushagarwal ankushagarwal mentioned this issue Apr 19, 2018
Merged
k8s-ci-robot pushed a commit that referenced this issue Apr 20, 2018
@k8s-ci-robot
Refactor tensorflow-notebook-image/Dockerfile (#689)
0ad94c4 
Remove a lot of bloat - install only the minimal set of packages
required to get started with ML. Any packages required can be installed
by the user in the notebook itself using pip install/conda install

Image size has gone down from 12GB to 3GB for cpu image

Having a lot of packages makes it very challenging to maintain them
because of version conflicts

Run everything as jovyan user - this enables user to run conda install
/ pip install without requiring sudo

Add comments on every step

Fixes #668
Fixes #37
Fixes #472
pdmack pushed a commit to pdmack/kubeflow that referenced this issue Apr 21, 2018
Refactor tensorflow-notebook-image/Dockerfile (kubeflow#689)
fc212db 
Remove a lot of bloat - install only the minimal set of packages
required to get started with ML. Any packages required can be installed
by the user in the notebook itself using pip install/conda install

Image size has gone down from 12GB to 3GB for cpu image

Having a lot of packages makes it very challenging to maintain them
because of version conflicts

Run everything as jovyan user - this enables user to run conda install
/ pip install without requiring sudo

Add comments on every step

Fixes kubeflow#668
Fixes kubeflow#37
Fixes kubeflow#472
Conflicts:
	components/tensorflow-notebook-image/Dockerfile
	components/tensorflow-notebook-image/build_image.sh
	components/tensorflow-notebook-image/releaser/components/workflows.libsonnet
k8s-ci-robot pushed a commit that referenced this issue Apr 21, 2018
@k8s-ci-robot
CP to V0.1 branch: TF notebook slimming, joyvan pip installs, and new…
b497fd5 
… gcr.io locations (#703)

* Refactor tensorflow-notebook-image/Dockerfile (#689)

Remove a lot of bloat - install only the minimal set of packages
required to get started with ML. Any packages required can be installed
by the user in the notebook itself using pip install/conda install

Image size has gone down from 12GB to 3GB for cpu image

Having a lot of packages makes it very challenging to maintain them
because of version conflicts

Run everything as jovyan user - this enables user to run conda install
/ pip install without requiring sudo

Add comments on every step

Fixes #668
Fixes #37
Fixes #472
Conflicts:
	components/tensorflow-notebook-image/Dockerfile
	components/tensorflow-notebook-image/build_image.sh
	components/tensorflow-notebook-image/releaser/components/workflows.libsonnet

* Update various images in kubeflow to kubeflow-images-public (#635)

Point them to kubeflow-images-public instead of kubeflow-images-staging

Related to #534
/cc @jlewi
Conflicts:
	bootstrap/Makefile
	bootstrap/README.md

* Migrate images to kubeflow-images-public (#695)

Related to #534
Conflicts:
	bootstrap/README.md
	docs_dev/images.md
	kubeflow/core/tests/tf-job_test.jsonnet

* Update the hub spawner dropdown for latest NB images (#697)
saffaalvi pushed a commit to StatCan/kubeflow that referenced this issue Feb 11, 2021
@k8s-ci-robot
Refactor tensorflow-notebook-image/Dockerfile (kubeflow#689)
ab1e781 
Remove a lot of bloat - install only the minimal set of packages
required to get started with ML. Any packages required can be installed
by the user in the notebook itself using pip install/conda install

Image size has gone down from 12GB to 3GB for cpu image

Having a lot of packages makes it very challenging to maintain them
because of version conflicts

Run everything as jovyan user - this enables user to run conda install
/ pip install without requiring sudo

Add comments on every step

Fixes kubeflow#668
Fixes kubeflow#37
Fixes kubeflow#472
elenzio9 pushed a commit to arrikto/kubeflow that referenced this issue Oct 31, 2022
@shannonbradshaw
Add shannonbradshaw to members list. (kubeflow#472)
8def894
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/jupyter Issues related to Jupyter help wanted priority/p3 Very low priority.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@jlewi @ankushagarwal