Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Maven images with prefetched local repo speeds up Maven build startup #130

Open
Tibor17 opened this issue Oct 27, 2019 · 7 comments
Open

Comments

@Tibor17
Copy link

@Tibor17 Tibor17 commented Oct 27, 2019

When you run Maven with empty local repo, it looks like Maven sleeps but it downloads plugins in the background. Having new images e.g. 3.6.2-jdk-14-prefetched would speed up the start up of the build. Just RUN the command mvn clean install and let the Maven to download the plugins when you build the image. Then push the image to dockerhub. Thx

@Tibor17

This comment has been minimized.

Copy link
Author

@Tibor17 Tibor17 commented Oct 28, 2019

@softbrada
Can you tell me why you disagree?
We in ASF Maven spoke about prefetched local cache and we can see it as a big mprovement.
I do not say that the current images should have the cache. I named the image 3.6.2-jdk-14-prefetched because it is a new one. All Dockerfile-s and images would stay as before.

@carlossg

This comment has been minimized.

Copy link
Owner

@carlossg carlossg commented Oct 28, 2019

@Tibor17

This comment has been minimized.

Copy link
Author

@Tibor17 Tibor17 commented Oct 28, 2019

@carlossg
I know what you mean but in our case we are not a user. We develop the Maven itself in Apache. So we were thinking about deploying Maven docker images with pre installed plugins (not a customer's dependencies). This would speed up the start time of the container or Maven build process actually.

So there was one idea in ASF to build images by us. And second idea was to ask you to support us with new additional fat images. The fat images would differ against the current ones only that the plugins would be preinstalled in the cache so the users would not wait too long when his container starts up.

And since Carloss' images on Dockerhut are so famous and the result so good that we don't want to compete.

@softbrada

This comment has been minimized.

Copy link

@softbrada softbrada commented Oct 28, 2019

@Tibor17
Firstly I want to say I am not a maintainer or a contributor to this repo however I am actively using these images from this repo which are very great.
But regarding your feature request I disagree because it îs not doable. At least not în The way you want it. I might be wrong although i have been using Docker for several years with several languages.
Let me explain
The way you want this to work îs that when you pull a image from dockerhub lets say 3.6.2-jdk-14-prefetched, you want this image to already have somehow magically all the dependencies you have in Project X which lets say îs the curent Project you are working on. And then when you use Project Y to have dependencies already installed în same image from this Project Y. That just seems like non-sense to me. Since no one can do that since no one has access to your own pom.xml files.
So a published image cant already contain that.
But you can achieve this în a several ways.( I am going to summarize few options wothout going into too much details)

  1. Clone this repo locally, create a hooks directory locally I same directory where the Dockerfile is located and add a "post_build" hook, add your script there and build the image locally with Project X and publish it în your own private registry. But when you want to use that with Project Y you need to build locally a New image with different name and publish it to your own private registry althouth the only difference îs that you have the dependencies already installed în The images themselves. I honestly dont see why you would publish 2 images with 1 GB each or more to a registry just to have those dependencies. Seems like a waste of space to me.
  2. Do mostly same thing as you did before but now instead of cloning the repo you just create a dockerfile locally and use either what @carlossg said or a mulți-stage build and build two different images for Project X and Project y as before and publish them to your private registry. Although you still end up with two huge images that basically just have the dependencies installed.
  3. The Best way to do it and avoid all of this non-sense with multiple images for each Project. Try and use docker-compose.
    And add în The docker-compose file în The volume list something like
${HOME}/.m2/repository:/home/root/. m2/repository

So everytime you build your images the volumes will use the host cache directory.
So regardless of how many projects you have you have a single image for all of them which does NOT have în its layers the dependencies installed but when you start the container the container will mount the volume and the installation will use the cache

Maybe I am having a different perspective on this feature request or maybe I am getting it wrong. I might not be up to date with latest Technologies and best practices. So please let me know if I am missing something or I am not correct about something. But to me the last option îs easier. You can mount that volume even without docker compose by using two commands. First the docker build and then with docker run but specify the volume too when you run the container.

What I suggest îs to keep reading the docker documentation and the readme of this Project. There îs quite a lot useful information.
On another note.. I really love this project. Hope all of this information will be useful.

@carlossg

This comment has been minimized.

Copy link
Owner

@carlossg carlossg commented Oct 28, 2019

we can post prebuild images but we would need to figure out what plugins to include and how to manage the combinations between base image and plugins

@Tibor17

This comment has been minimized.

Copy link
Author

@Tibor17 Tibor17 commented Oct 28, 2019

we can post prebuild images but we would need to figure out what plugins to include and how to manage the combinations between base image and plugins

Very good point.
A separate image with plugins was what I had in my mind too but one of my colleage pointed out that Maven may change the set of versions of plugins between releases. It does not happen very often, e.g. next time it will be in the version 3.7.0 after long period.

to figure out what plugins to include

yes, so we were facing two options:

  • the set of plugins per Maven release with certain versions defined by the Maven dist, or
  • the range of versions of the plugins for the build lifecycle.

The answer also depends on e.g. the size of local cache.
A simple experiment (often used command mvn clean install) on trivial project shows that the size of the local cache is only 12 MB and this way we speed up the build in 50 seconds.

I guess the size is not big but the ping pong between Maven and repository causes that the user has to wait. It is funny situation on conferences when the presenter starts a container and the audience is wating one minute because the console shows pretty nothing and the Maven downloads basic plugins in the background. ;-)

Regarding the concrete plugins, it should be whole spectrum related to the lifecycles and the plugins would be mostly these: clean, resources, compiler, surefire, jar, war, ear, ..., install, maybe the deploy and the site plugin as well.

@carlossg
If you agree, I would open a public thread on the Maven maling list for developers and you, and our collagues who are involved in this activity would join us. I hope you will attend the discussion too.

@carlossg

This comment has been minimized.

Copy link
Owner

@carlossg carlossg commented Oct 29, 2019

you can have one or multiple pom.xml that gets executed as a multi stage docker build then keeps the plugins.
That's probably the easiest way to download the plugins without having to worry about versions

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.