Consider offering an expanded image #36
Comments
|
Hey @KyleAure, that is a very good comment and a long-standing discussion between users of database images. Some people want the smallest possible image while others want the fastest possible start-up time (which I was always a fan of myself). There is actually a simple trick to get rid of that uncompression time, and I promised @Sanne to document it properly (#53), which is to use a volume between runs, if possible, of course. If the database data files are already present inside the container, the container will not extract them again: oci-oracle-xe/container-entrypoint.sh Lines 340 to 349 in 2a9fd71 However, one would have to have such a volume ready to mount, which often can't be done or only with some extra steps. What I really would be interested to know is whether this would work for your test environment as well, or whether that's not good enough? |
|
I think having a mountable database would be a good solution for some users. But you are correct that it won't be an option for most depending on the CI/CD system they are using. In our case it might be possible, but would greatly increase the changes of one of our test suites affecting the next if they don't properly cleanup the database before they are finished. Honestly, I think both images would be valiable in their own right. If we look at the image sizes from the past: I created a docker image that has the database aleady installed using the 18-slim image as the base image and the size is I'd be willing to help contribute this to the this repo if you think it's a good idea. |
KyleAure
changed the title
Would that imply that state is persisted? We mostly use these for integration tests: it's actually important that the DB is in pristine state at each start. But I second @KyleAure 's request. It's of course nice to have small images, but bootstrap times are more important in such contexts. |
|
@Sanne yes, volumes persist data between container runs. I am unsure if there is a way to mount data (besides a straight copy) to a container without persisting it afterwords. |
|
I may be going in the wrong direction but for what it’s worth, I used testcontainers and an embedded oracle xe image whilst integration testing my code. It did take up to 30 seconds at one point but I guess that’s manageable in a CI/CD set up. |
|
Yes, for CI the fastest starup is wanted. The docker images are stored uncompressed, but compressed when downloading. So as long the size to download will not be significantly larger, the benefit would be that the uncompression will be done only once - by docker, which is usually faster because it is optimized for max. speed, and any subsequent image startup (of already downloaded/decompressed image) will be fast. |
|
I finally came around to this and did a couple of tests yesterday thanks to (@mvorisek) bringing this up in another conversation over at #124. As a test source, I took the I think there are no big surprises here as that size is roughly what Kyle pointed out above as well. The compressed sizes of the layers are equally within the expected area. The expanded image is almost as small compressed as the non-expanded image, which should be the case given that the compression method of the database data files is similar:
Additionally, pull tests show that the ~15 - 20 seconds uncompression time moves into the
The important point here is that from a timing perspective nothing will change for someone who is pulling an image and starting a container once, as the uncompression phase just moves from the container startup to the image pull operation. If you would like to test the image, you can currently get it via |
|
Thank you very much for looking into this feature request. The Yes, the uncompression time is moved into the
moved - of course only once in the expanded case, then the boot time is about twice as fast |
|
Awesome, thanks a lot! For bootstrap times the new images definitely help: My experiments: Connected in ~5 seconds. Previously: Connected in ~16 seconds. That's very nice! |
|
Great, thanks both! I will wrap up these images then (including doc, etc.) and add them to the repo. Stay tuned! |
|
Awesome - as soon as they're avaIable I'll switch all Quarkus users to these 👍 |
Signed-off-by: gvenzl <gerald.venzl@gmail.com>
Signed-off-by: gvenzl <gerald.venzl@gmail.com>
|
Alright, so I got to take a look into this and there are now These faststart images have a new third layer added in which the uncompressed database files are sitting. The nice thing about this approach is that the other two layers can be reused from the non-faststart version. For example, if someone has already the Doing some quick tests on my end here with a relative fast internet, it takes about 50 seconds to pull the faststart image when the non-faststart equivalent is already present: Comparatively, if no images are yet present on the system, it takes me about 1min 30sec to pull the image, which is about the same as it was in the test from last week above: The reason for that is because the container runtime is pulling the layers in parallel anyway, so there is no sequential time build-up. By the time the larger layer has been pulled, the smaller one was also already downloaded. However, this is, of course, highly subjective to one's internet connection and host machine processing power. Here again are the pull times from last week's test with the two layers image: Another benefit of sharing the first two layers is that if the faststart image flavor is already on the system, pulling the non-faststart image becomes a no-op: I hope this is what you were looking for. |
|
This looks really great @gvenzl Thank you so much for looking into this! |
|
Of course, thanks for bringing it up and, of course, for using these images! |
|
Awesome, many thanks @gvenzl ! |
|
Great, thank you! I'm curious, do you have some timings that you could share of how long tests took before and after? |
|
It would vary significantly, but for example in Quarkus "core" there's at least three different Maven modules that need integration tests to execute in connection to an Oracle RDBMS on each CI run, and we prefer not reusing the instance across modules. So right there we'll save about 1m per CI build. That's very welcome news, as CI build times are a problem - even worse for people testing locally. End user's workflow - assuming they use Oracle - would imply trigger starting one of these each time a dev-mode session is initiatied, and once (at least - depending on the project structure) for each build. On my particular machine I'm saving ~18seconds each time there's such a need; I would imagine savings are larger for each user. But more than about the exact amount of seconds, I'd say it helps with the feeling of smoothness and bring confidence to people using it - it's just much nicer to work with it :) A quick start has a psycological impact on making it feel lightweight rather than "bloated" - even if there's of course no corelation to its actual memory consumption and runtime efficiency. For reference, I really like the PostgreSQL container image; it starts in about a second which makes it such a pleasure to work with. Mentioning it in case you get bored :-D |
|
Thanks @Sanne, appreciate the feedback and fully understand. Regarding
I will send all the people complaining that the image is too bloated because "it's too big" your way from now on! ;) |
Currently, at container startup the database goes through a decompression phase at:
oci-oracle-xe/container-entrypoint.sh
Lines 131 to 139 in 977b00e
I have noticed this decompression phase takes anywhere from 15 seconds to a minute depending on the machine the container is started on.
Myself, and other developers that use these containers for testing, would likely accept the trade off of a bigger image, to reclaim performance at container startup.
The text was updated successfully, but these errors were encountered: