Refactor images

[Silex]

@gonewest818, @DamienCassou, @jgkamat

For your information, I'll merge this soon. The main changes that might concern you are:

• WORKDIR is gone.
• Don't install in /opt/ anymore.
• -slim image are build differently.

That was made so -alpine images will come soon.

This PR is mainly so you can protest against a change for whatever reason you might have, and that we can find a solution together. Without any news from you in the next days I'll just go and merge it.

[jgkamat]

Looks good to me, I like all the changes so far and I don't think this will break any of my builds. Nice! :)

[Silex]

Hum. I just realised multi-stage builds allow me to shave ~170MB from the image.

Basically, remove the source from the layers! The trick is to copy the source in an early stage, build there and then copy the build artefacts.

So, looks like we are back to /opt/emacs in the meantime because shaving such a large amount is nice.

[gonewest818]

I left a few comments in the code... thanks

[gonewest818]

If you're going to do this, then perhaps you should keep the emacs sources in the first stage & tag that image separately for developers who want to inspect exactly how Emacs was compiled. The way docker manages layers there is no additional cost because, as you've discovered, the rm -rf /tmp/emacs didn't actually cause the image to be smaller anyway.

[Silex]

Interesting, do you think there's a need tho? I will already have quite a lot of tags... If things go well the -alpine version will come, and if I follow your idea this will add another variant, so we'll have:

• dev (with /tmp/emacs)
• normal (without /tmp/emacs)
• slim (without /tmp/emacs and without -dev packages)
• alpine (from alpine, without /tmp/emacs and without -dev packages)

If you look at https://github.com/Silex/docker-emacs/blob/refactor/README.md, the list is decently long... add 24.3 and 24.4, then double that list because of -alpine and -dev. This becomes a wall of tags :-) But maybe that can be solved with better formatting.

[gonewest818]

Well the other alternative is to encourage the “slim” builds, which are guaranteed to be even smaller than this. Just dev, slim and alpine.

[Silex]

@gonewest818: I like your idea! tags 25.3-dev, 25.3 and 25.3-alpine

[gonewest818]

Assuming this first stage is identical to the first stage of the non-slim build, then you don't need this repetition. You can use a single Dockerfile and just run the build stages individually and tag them individually.

[Silex]

True, but then you lose the "1 image = 1 dockerfile" spirit that makes the README a great way to quickly visualize what is going on.

Except if you meant COPY --from=silex/emacs:25.3 /opt/emacs /opt/emacs/, but I don't think this is allowed. I need to try tho.

EDIT: woa, COPY --from=silex/emacs:25.3 is allowed! That's so nice!

That means we can build once and then derive all the other images from the root one. I'll need to think about it.

[gonewest818]

Yes, that does work and in particular for the situation I just commented on, where the builder images are named differently to keep the namespace clean.

[gonewest818]

I'd like to see a variant without Cask and python for people like me who don't want or need them in the image.

[Silex]

That'd make 5 variants! :-) Wait no, 8 variants! Each of dev, normal, slim, alpine would need a -cask variant.

Right now it feels a bit too much, especially since python/cask does not really add much MB or hurt when present.

Thank you for the feedback/review! I'll ask what other thinks about your points on the issue itself.

[gonewest818]

I’ve seen projects separate the build images and the “run” images into separate namespaces so that the naming isn’t so cluttered.

So for instance most users would never be concerned with the dev images:

Silex/docker-emacs-builder:25.3 (this is the “dev” image)

But they do select either the emacs-only images

Silex/docker-emacs:25.3 (this is “slim”)
Silex/docker-emacs:25.3-alpine (“slim+alpine”)

Or the ones with cask too

Silex/docker-emacs-cask:25.3 (this is “slim+cask”)
Silex/docker-emacs-cask:25.3-alpine (“slim+alpine+cask”)

You might find the Alpine images are small enough that adding python is significant.

[Silex]

Again I like your idea! It makes sense. I'll start by creating all the images in this repo and then split the repo in 3 when the time is right.

Thank you a lot 👍

[Silex]

@gonewest818 suggest we have -cask variants (with or without cask/python) and also -dev version where you have access to /tmp/emacs, allowing you to re-run make, etc.

What do you think of these ideas?

[Silex]

Okay, for now I see several possibilites.

One namespaces, all possibilities:

image	dev	cask	needed
silex/emacs:25.3	no	no	yes
silex/emacs:25.3-cask	no	yes	yes
silex/emacs:25.3-dev	yes	no	no
silex/emacs:25.3-dev-cask	yes	yes	yes
silex/emacs:25.3-alpine	no	no	yes
silex/emacs:25.3-alpine-cask	no	yes	yes
silex/emacs:25.3-alpine-dev	yes	no	no
silex/emacs:25.3-alpine-dev-cask	yes	yes	no

Multiple namespaces, all possibilities:

image	dev	cask	needed
silex/emacs:25.3	no	no	yes
silex/emacs:25.3-cask	no	yes	yes
silex/emacs-dev:25.3	yes	no	no
silex/emacs-dev:25.3-cask	yes	yes	yes
silex/emacs-alpine:25.3	no	no	yes
silex/emacs-alpine:25.3-cask	no	yes	yes
silex/emacs-alpine:25.3-dev	yes	no	no
silex/emacs-alpine:25.3-dev-cask	yes	yes	no

One namespace, only the needed ones:

image	dev	cask
silex/emacs:25.3	no	no
silex/emacs:25.3-cask	no	yes
silex/emacs:25.3-dev	yes	yes
silex/emacs:25.3-alpine	no	no
silex/emacs:25.3-alpine-cask	no	yes

I think I prefer the last one for simplicity reasons. My rationale is that if you want the -dev image, having cask will not hurt. If you want the -alpine images, then you'll never want its dev version (because size). And the problem with multiple namespaces is that it requires interlinking between the hub repositories so basically 3 READMEs. Btw the normal images are actually the previously named slim ones.

[gonewest818]

I still prefer separating the namespaces to help guide users to the right builds.

Also, personally I don't care about the presentation of images on docker hub. I think github is a more effective way to promote the images, allow for better "discovery", provide more detailed documentation, etc. That's why in earlier commits I didn't worry about reorganizing the Dockerfiles and markdown in ways that wasn't perfectly compatible with the docker hub site.

Ultimately though, I understand my priorities are different than yours, perhaps dramatically so, and fully acknowledge your approach is fine for normal use.

About the dev-alpine variant -- I think it would be useful to have base images that contributors can use as the starting point for other build variations. Compiling --without-x would be an obvious one. But looking forward I'm also noticing the chatter on the emacs-devel mailing list that a new "portable" dumper could be merging in. I would expect that feature to appear as a compiler option for some period of time until the bugs are ironed out. So laying down the foundations to afford devs easy access to the builder images seems like an investment worth making.

[Silex]

Ultimately though, I understand my priorities are different than yours, perhaps dramatically so, and fully acknowledge your approach is fine for normal use.

I need to think more about it, because if it's all automated it's not really a problem to have 3 repos, especially since today I discovered I can automate the README update on the hub with a dummy branch.

About the dev-alpine variant -- I think it would be useful to have base images that contributors can use as the starting point for other build variations. Compiling --without-x would be an obvious one. But looking forward I'm also noticing the chatter on the emacs-devel mailing list that a new "portable" dumper could be merging in. I would expect that feature to appear as a compiler option for some period of time until the bugs are ironed out. So laying down the foundations to afford devs easy access to the builder images seems like an investment worth making.

Ah, good to know the portable dumper discussion is back from the dead... we'll see where that goes. Your dev-building-blocks idea is very interesting, it immediatly got me thinking into -src variants where emacs is not even build or anything, just with the correct source at /opt/emacs. Also maybe some kind of -onbuild images where you can specify the configure flags using build args, or a configure-flags.txt file convention (like ruby/python onbuild images, etc).

EDIT: hum, maybe -onbuild is not such a good idea docker-library/official-images#2076

[DamienCassou]

What is the status of this PR?

[Silex]

@DamienCassou: brewing. Most commits here will be rewritten.

When it's done, we'll have "normal" (what is called slim at the moment), -dev and -alpine variants, and with or without cask.

I'm still debating wether alpine/cask images go in a repository on their own. At the moment I'm working on including a "patch" phase to the emacs source in order to properly build the alpine images, and how the final images.yml should look, etc.

[Silex]

Alright. To solve the design issues I did some README-driven development 😄

I arrived at two possible ways:

1. https://github.com/Silex/docker-emacs/blob/7dbc13b8695d745ff280d48e1a95e99a025bd522/README.md
2. https://github.com/Silex/docker-emacs/blob/b6d561f876fb1cdaf9e3060d27d0fe718ce1f58f/README.md

Here are the tradeoffs for option 1:

• Simpler. Just the most common needs. Easy to understand the options.
• Hard to add more variants/tags because we'd quickly lean toward option 2.
• Based on the premise that Cask is mostly needed for developpers, so it makes sense that only the -dev images contains it. If Cask users want -alpine or "normal" variants then this will be complicated.

Here are the tradeoffs for option 2:

• No one would be able to complain that he doesn't have what he wants (Cask users happy).
• Too many choices, can be confusing.
• Many images will be useless because no one would use them.
• Adding more tags/namespaces is easy.

I'm slightly in favor of option 1 because simplicity wins in my book.

@gonewest818, @jgkamat, @DamienCassou: voice your opinions 😉

[jgkamat]

I personally prefer option 1, option 2 is too overloaded in my opinion (and confusing which one I'd want, too many choices :P).

I don't think having missing cask on some images is a big deal, since it only takes seconds to install.

In short, I don't think the little benefit gained from having every image possible is worth the build complexity and the information overload when picking images. If someone wanted that much specificity, they should just build their own image FROM this one :P

[Silex]

Okay, I'm almost merge-ready.

Unfortunately, one of my plan failed miserably, which was to be cute and have 25.3 to copy from 25.3-dev, like we see here:

docker-emacs/master/xenial/Dockerfile

Line 27 in bba48df

COPY --from=silex/emacs:master-dev /usr/local /

I had to create a depends: key in the yaml, refactored it all in ruby so one job can build multiple images, etc... but Travis uses docker 17.09.0-ce which does not allow COPY --from another image like in 18.03.0-ce:

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED                  SIZE
new-image           latest              84ca91668fd0        Less than a second ago   1.24GB
silex/emacs         master-dev          84ca91668fd0        Less than a second ago   1.24GB
...
Step 5/6 : COPY --from=silex/emacs:master-dev /usr/local /
invalid from flag value silex/emacs:master-dev: manifest for silex/emacs:master-dev not found

Which basically means I did pretty much all this for nothing 😅

I'll keep this work for future refactoring in some branch, might be useful some day. I'll just keep the images.yml and update script, the rest is probably going back to bash scripts.

EDIT: actually COPY --from works but the image has to be on the docker hub already, so I could push it before building the other one. That said, I realised it also runs into other problems because as the repository is hard-coded it prevents forks from pushing their own images. This depends: idea looked cute but is more trouble than it's worth.

[Silex]

Merging as soon as the travis-build is successful 😄