satra/dandihub:e3b52ec0 image too big?

[yarikoptic]

trying to start a tiny issue on hub. http://hub.dandiarchive.org/hub/spawn-pending/yarikoptic is either taking too long (minutes)

NB it did finish eventually

docker pulling on a hefty/speedy smaug shows that some layers are huge (>=600MB)... and actually I even ran out of space
(on smaug)

failed to register layer: ApplyLayer exit status 1 stdout:  stderr: write /opt/conda/envs/ros3/lib/python3.9/ensurepip/_bundled/setuptools-49.2.1-py3-none-any.whl: no space left on device

not entirely sure why yet since

$> df /opt -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/md0        147G  125G   15G  90% /

https://hub.docker.com/r/satra/dandihub/tags?page=1&ordering=last_updated says it is 3.6GB .

I wonder if it couldn't be minimized -- sounds like too big FWIW

[yarikoptic]

FWIW, was eyeballing free space while redoing the pull -- never went below 7GB still available. So not entirely sure what is going on here but I reliably can't pull on smaug.

[satra]

the image has multiple environments. a normal one with dandi, an allensdk environment, and a streaming environment. also it's built on the jupyter datascience base, which includes Python, Julia, and R.

[yarikoptic]

FTR: and the

jupyter/datascience-notebook          016833b15ceb        1fb531b11821        3 months ago        3.91GB

so it is huge already but at least "runs" for me

[satra]

i think that's the uncompresed size. the uncompressed size for dandi is:

dandiarchive/dandihub latest d60f907f233b 31 hours ago 12.8GB

[satra]

so we should do some tweaking and optimization.

[yarikoptic]

FWIW (I am not looking to switch to singularity, just to establish some kind of a bound): I

singularity pull "docker://dandiarchive/dandihub:latest"

to assess what takes up space inside, I get

$> du -scm dandihub_latest.sif
2817    dandihub_latest.sif
2817    total

so my guess that majority of size might be coming from all the FS layering of docker, not really the content (not that we likely should be able to shrink it too).

Just FTR:

FWIW couldn't exec that image ;-/

$> singularity exec dandihub_latest.sif bash
FATAL:   container creation failed: mount /mnt/btrfs/scrap/dandi->/mnt/btrfs/scrap/dandi error: while mounting /mnt/btrfs/scrap/dandi: while getting mount flags for /mnt/btrfs/scrap/dandi: while searching parent m
ount point entry for /mnt/btrfs/scrap/dandi: no parent mount point found
$> singularity --version
singularity version 3.5.2+ds1

but could with clean env, no binds

$> singularity exec -e -c dandihub_latest.sif bash
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

Singularity> ls /
bin  boot  dev  environment  etc  home  lib  lib32  lib64  libx32  media  mnt  opt  proc  root  run  sbin  singularity  srv  sys  tmp  usr  var
Singularity> ls /opt
conda  julia  julia-1.5.3
Singularity> ls /opt/conda/envs/
allen  ros3

[satra]

i'm going to close this. the image is a little smaller now. but each conda environment also installs itk a dependency for nwbwidgets (which is 1G in size uncompressed). so that alone adds to it. i've revamped the creation to minimize the size per layer, but they are still large.