CRIU enabled container image #18229
Comments
Unfortunately, there is an issue with our docs. We only package criu with the ubi8 and 9 images. We have a custom version that we release on UBI that includes unprivileged and random UID support and other bug fixes. We are working on releasing a similar version for docker images.
I tried the following: It seems to be there for me. |
For evaluation I recommend using the UBI8 or 9 images. If you need ubuntu then you can copy the dockerfile you linked. But, I recommend updating the criu repo to https://github.com/ibmruntimes/criu/tree/0.40.1-release. And downloading the latest binaries here: |
|
@fipro78 Were you able to make any further progress? |
|
@tajila Unfortunately I am running from one issue to the other. After your suggestion to use the UBI8 or UBI9 images, I managed to build the images, but starting the application still fails at checkpoint creation. Short interlude to my setup:
As the automated image creation via Maven build doesn't show the error, I started trying to build manually. The first thing I came around is that I need to use privileged containers if I want to create a checkpoint.
After some googling I found these two: I tried to build from the Linux based devcontainer, and even from within a WSL2. But always the same error. Most of the time it sounds like the issue is the usage of Docker and the solution is to use Podman. So I tried to install Podman. But as I am behind a corporate firewall, I am somehow unable to get Podman to work. I always get errors related to proxy or DNS resolution. Actually I am a bit stuck right now. The usage of Docker on Windows seem to be broken at the moment. To test Podman I need to first find a computer that is not necessarily behind a corporate firewall. Of course that should be the easier task when I find the time to test on my private computer. But not ideal if I want to promote this inside the company. And for my build setup with Maven and the Docker Maven Plugin, I expect it also to fail, as it seems that I need Podman for the image creation with automatic checkpoint creation. Anyhow, I don't give up, I probably just need some more time to get something working. But it is of course unfortunate that the Docker usage still doesn't work. Surely not your fault, just a thought. :) |
|
@tajila Sorry for the delay. I had some troubles with my environment and did some CRaC testing in parallel. I have created a small setup with the OSGi application and some scripts to build the container image via the three step process. Trying to create a checkpoint inside the container from Windows CMD via Docker or Podman and from within a WSL fails with this error: The log contains this info at the end: Running the script via Podman inside the WSL fails with this error: I have no idea what is going wrong. Is it an issue on Windows? Is it not possible to create a checkpoint for my app? Is something else missing? The attached zip archive contains four folders. The criu_ folders contain the Dockerfiles and the scripts to build the containers. I tried to run the scripts from Windows directly and from an Ubuntu 20.04 WSL. Maybe someone can test this on Linux or Mac? Or maybe someone sees what I have done wrong? P.S. the crac_ folders contain the same example using Azul CRaC. There I can create the checkpoints in some scenarios, but the restore fails. |
|
@fipro78 Just to confirm this is where the checkpoint is being triggered, https://github.com/fipro78/osgi_deployment_options/blob/criu/org.fipro.osgi.benchmark.criu/src/main/java/org/fipro/osgi/benchmark/criu/BenchmarkCRIUSupport.java ? |
|
If there are any active TCP connections at the time of checkpoint |
|
@tajila Yes, that is the service that triggers the checkpoint creation. There shouldn't be an open TCP connection. At least none I am aware of. But maybe it is the console that opens one. Not sure. I have added Using Docker: In the log: Using Podman (rootless): I actually tried it following the approach described in this article: https://blog.openj9.org/2022/09/29/unprivileged-openj9-criu-support/ Reading the InstantOn limitations and known issues I noticed the section about Running without the necessary Linux capabilities. The error looks exactly the same as mine when trying to build with Podman. So for testing I tried to create the checkpoint with Docker/Podman and I now tried it in a WSL2 with:
Not sure what to try out next. |
|
@fipro78 thanks for the test. I took a look at the contents of
When CRIU restores a checkpointed process the original PID and TIDs of the process must be available. When we run in containers we are in a new (empty) PID namespace, so processes tend to get very low PIDs/TIDs. This is a problem for CRIU because on restore unless you pay close attention to every process that starts and their ordering it's likely that at least some of the required low PIDs/TIDs are taken. We work around this by invoking a dummy command 1000 times so the Java process can start with PID/TIDs >1000, which on restore are very likely to be free. In unprivileged mode we can't easily re-attach std out/err/in to terminals, we need to tie stdin to /dev/null and redirect stdout and stderr to files, and those files need to be present at the same path on restore. The easiest solution is to just dump them to the checkpoint data dir. Elsewhere I've made changes to make sure is available at the same path on restore. Here we need to pass For Docker we first need to make sure we are using Docker 23.x or later. The 20.x versions do not support CAP_CHECKPOINT_RESTORE. Second we need to add For Podman we first and foremost have to run as root. In order for options like As you can see there are a few extra steps in the process, and currently they're not well documented. In the near future we're hoping to have some helper scripts and commands to make life easier. Most of this works pretty well in Open Liberty because we do have the necessary scripts and helpers there, but the same experience isn't yet available in Semeru images, but will be. Also note that rather than copying the checkpoint data from the first container back to the host, then building a new image and copying the data in, you can use a slightly different method: you can commit the original container and use the |
|
@ymanton Attached the updated example: criu_test.zip I will try to include this into my GitHub project, so all the Java deployment options for OSGi applications are collected in one place. I still have one question (at least, probably a lot more when it comes to details ;) ). Is it possible to create a checkpoint using the Right now I needed to create a devcontainer to develop the code for the checkpoint creation, because my host system is Windows. BTW, with your suggestions I also got the image creation working on Windows with Podman. I had to configure the machine as rootful, as on Windows there is no sudo concept. |
Currently no, but I think we are looking at this. @tajila can probably elaborate. |
Yes, we are currently working on adding some support for this. We should have something in Q1 2024. |
|
@tajila |
|
@fipro78 Sorry for the late notice. If you try a recent version of OpenJ9 (https://github.com/ibmruntimes/semeru17-binaries/releases/tag/jdk-17.0.11%2B7_openj9-0.44.0-m2) you'll be able to use This year we are planning broader support for checkpoint/restore on Semeru so Ill can keep you posted on new developments. |
|
@tajila Then I tried it with the OpenJ9 JDK and JRE. I used one of the available containers, but in both cases the jdk.crac classes can not be found. Dockerfile start.sh Service class that creates the checkpoint using the org.crac API When I try to run the example with an OpenJ9 I get the following exception (I only see it because I locally patched the org.crac jar to print out the exception, otherwise it is catched silent): I checked the Java version in the container, and got the following output: According to your message, I would assume it should support the org.crac API. But it looks like it doesn't. I downloaded the archive and also do not see the jdk.crac module in there. Have you dropped it from the 0.44.0 release? Is there anything I have to do additionally to make the org.crac API support to work? |
|
@fipro78 Thanks for your response. I dont have access to your example so I made a little one of my own: Dockerfile start.sh CracExample.java Then So it seems to work for me. Perhaps you can send me the .jar file in your example. |
The crac support is in |
|
@tajila The example is actually some work in progress, because there are multiple levels to fix before it can be published.
As I said, it is work in progress, so I have not published it yet. But the goal is to publish it together with a blog post soon, so everybody is able to follow and reproduce things. I have attached the example so you can try. Hope that already helps. What I noticed a few hours ago is, that the concept about registering a Resource vs. the Hooks for closing and reopening resources, is different in CRaC and OpenJ9 CRIU. With CRaC I can register a Resource where I have access to it. For example in the Jakarta-RS Whiteboard at the location where I have access to the Jetty server instance. It is decoupled from the place where the checkpoint creation is done (in my example the immediate component). With the OpenJ9 API it seems the hooks need to be registered where the checkpoint creation should be performed. At least I have that assumption now, as I tried to get it to work for a while, similar to the Resource, but the hooks need to be registered on a Anyhow, either there is a classloading issue somehow, or the way how Resources are handled does not correctly work. If you have any idea, let me know. I am currently trying to polish the example as much as possible. Then I can push it to my repo and then extend it step by step. |
|
@fipro78 I was able to reproduce the issue, still investigating why it can't find the jdk.crac class.
No, there is no restriction on where, or when the hooks can be added so long as it is added to the CRIUSupport instance before checkpoint creation is done. So in that regard, it is identical to CraC. One thing that is unique to J9 is that you have the ability many CRIUSupport instances, but that doesnt mean you necessarily need many. You could create a singleton instance (similar to CraC) and make it accessible via an API or
Can you please elaborate on this? I hope to resolve the jdk.crac issue by EOW, but either case Ill post an update here. |
|
@tajila I also noticed a possible threading issue. OSGi and the services is a multi threaded application. Maybe this is causing issues? I also try to use the Openj9 API in the example. But there I get again the error I will try to polish my example so that it is easy to look into it. I really don't get what is happening, as I did all of the steps you mentioned on the way. |
|
I did another local modification to the org.crac API additionally to printing out the exception. The org.crac API uses Reflection to find the real crac implementation classes. It uses The ClassNotFoundException persists. Only the stacktrace is different then. With With BTW, if I look at the |
|
@tajila the exceptions disappear. This is not necessary for the OpenJDK CRaC implementation. To be honest, I am not sure why the additional packages are resolved as system packages in the OpenJDK and not in OpenJ9. But maybe it gives you a hint. Or you tend to say it is not an issue of OpenJ9 and needs to be documented in some other place. But now I get the following exception: The created criu.log contains the following entries: I am using Docker inside a WSL on Windows. I will now also try to run it with Podman. But I need to setup everything again. Just wanted to update you with my observations. I added the updated example with the systempackage modification, in case you want to test it. |
|
@ymanton Can you please help Dirk resolve the CRIU errors.
Ill take a look at this. We put jdk.crac in java.base which is loaded by the system loader. Perhaps this is the key difference between the openJDK version. In which case I would be fine with changing it to match OpenJDK. Another difference, is the jdk.crac package is not exported automatically due to compliance reasons, so this is why it doesnt show up in the module-info.class. We dynamically export it at rutime. |
|
OpenJDK
This seems the cause. |
I looked at the example. The checkpoint fails because CRIU is being invoked in privileged mode and the user in the container doesn't have sufficient privileges for that. I can't see the application code, but either the application or the OpenJ9 CRaC layer has to call |
CRAC does not, but we can add |
|
I just tried this. Now the error is I tried to run with Docker in a Ubuntu Linux WSL. BTW, the same error I get when I try to use the CRIUSupport API directly. My example is available here: https://github.com/fipro78/osgi-jakartars I am still writing on the blog post and on the example for the CRIUSupport API. If you want to try out the example, you first need to run Hope it helps to identify the issue. |
|
@fipro78 your updated example works for me. What version of Ubuntu are you using in WSL? I assume 22.04? What is the kernel version ( |
|
|
I have updated my example and added the openj9 branch to see if using the I tried both variants (OpenJ9 with CRaC and OpenJ9 CRIUSupport) on Windows host using Podman, Windows Host using Docker and in Ubuntu WSL using Docker. I currently don't have a WSL with Podman. Need to check that. Using the scripts in the checkpoint_container folder it should be easy to build the containers for checkpoint creation Windows + PodmanOpenJ9 CRaC Support Result in criu.log: OpenJ9 CRIUSupport Result in criu.log: Windows + DockerOpenJ9 CRaC Support Result in criu.log: OpenJ9 CRIUSupport Result in criu.log: Ubuntu WSL + DockerOpenJ9 CRaC Support Result in criu.log: OpenJ9 CRIUSupport Result in criu.log: The errors are always the same. But I don't know what is wrong. Have I missed something? Is something not correctly configured? |
|
@ymanton / @tajila BTW, one question to the IBM Semeru containers. When I try to create a new directory I get a It only works if I call in the Dockerfile before. Is this by intention? I haven't found any clue about this anywhere. And for the images on DockerHub it is not necessary. At least not from the description on the page. |
|
@fipro78 I'll try via WSL today and get back to you. Just so I understand what you've tried, "Windows + Podman" and "Windows + Docker" are the native Windows versions of Podman and Docker, both using WSL2 under the covers? And "Ubuntu WSL + Docker" is the Linux version of Docker, installed from the Ubuntu repository inside your Ubuntu WSL2 environment?
The Semeru images based on UBI 8 and 9 specify a non-root user ( For the UBI-based images you can do |
|
Thanks for the clarification. And yes your assumptions are correct. |
|
Inside a fresh Windows VM I installed WSL: Inside the Ubuntu WSL env I created a user when prompted, installed Docker, added myself to the and ran build script; the output was: Ignoring the Your comments had I'll try Podman for Windows next and let you know how that goes. |
|
As mentioned in the comment above I have pushed my example to github to make it easier to see the actual code etc. But interesting that it works on your machine and I am not able to create a checkpoint on different windows 11 machines. |
|
OK thanks, I see the repo. Can you check the seccomp state inside your containers? The You can check as follows: By default you will see near the bottom of the output, which means seccomp is enabled. If you do |
|
Thanks for the pointer. I actually already opened a ticket for the seccomp issue, but totally forgot about it. :( I am running my WSL2 with
Does it really fail or do you just see the tty error, but the application is starting? Because I do not see the error. Ok, found out myself. If I start the container with And I solved the issue on using the OpenJ9 And one (hopefully) last question: |
CRIU's "unprivileged" mode still requires some privileges unfortunately. CRIU's default privileged mode requires root, and supports checkpointing and restoring processes that use many different kinds of kernel features. CRIU's unprivileged mode does not require root, but it still requires a handful of capabilities. In unprivileged mode CRIU will avoid trying to checkpoint/restore processes that use certain features that we know can't be supported, and for others it will try any way and expectedly fail, so in that respect it only supports checkpointing a subset of programs that privileged CRIU can support. In the JVM we've changed a few things to allow us to reduce the number of capabilities we need to grant CRIU; Seccomp is a different story. Seccomp allows container runtimes to block certain syscalls, even if those syscalls are allowed to be used by non-root processes outside of containers. CRIU uses certain syscalls that are allowed to be called by non-root processes but that most programs don't ever use. Some container runtimes block such syscalls in their default profile to be conservatively safe. If you want keep seccomp enabled you can create a seccomp profile that allows the syscalls CRIU needs and keeps the rest disabled. You can take a look at https://blog.openj9.org/2022/09/29/unprivileged-openj9-criu-support/ for a discussion on that topic, and here's an example seccomp profile that we've used in the past: https://github.com/eclipse-openj9/openj9/files/8774222/criuseccompprofile.json.txt |
|
For your information: I hope I have collected and written about all the information you provided correctly. If you think something is wrong or should be adjusted, please let me know. Thanks a lot for your support over the last months! |
|
@fipro78 Thanks for writing the blog, looks good. Just a few things: While its not stated in the docs, OpenJ9 does support OpenJ9 encourages minimal set of caps when restoring an image. So at checkpoint The big benefit with our approach is that we are not running with Overall, I think the blog does a great job in highlighting how to get started with both technologies and what to expect when using it. One aspect that is not discussed is how the technologies will perform in production environments. OpenJ9 has added diagnostic and tuning capabilities to address some of the pain points that will be encountered in production environments. Here are some aspects:
So while fast start is crucial for checkpoint/restore, it is important that there is support to ensure that the checkpointed image can be restored on a new system as though it was started there originally. We are building out this support and looking for more use cases to identify areas for improvement. |
|
@tajila
That is interesting. I tried it but I got |
Yes, that is correct |
As @tajila offered, I raise a question about OpenJ9 CRIU here. My tests are based on container images to compare container size and startup behavior. I noticed that
ibm-semeru-runtimes:open-17-jre-focalibm-semeru-runtimes:open-17-jdk-focalcontains the openj9.criu module, but there doesn't seem to contain criu support.
ibm-semeru-runtimes:open-17-jre-jammyibm-semeru-runtimes:open-17-jdk-jammydoesn't even seem to contain the openj9.criu module
ibm-semeru-runtimes:open-17-jre-centos7ibm-semeru-runtimes:open-17-jdk-centos7contains the openj9.criu module, but there doesn't seem to contain criu support.
For the
-centos7and the-focalcontainers I get the following exception on startup if I try to execute code that usesCRIUSupport:Reading the CRIU support page I assumed that CRIU support is available in the Semeru container images.
Why does the
-focaland the-centos7images contain theopenj9.criumodule, but the-jammyimages doesn't?Why is the criu support not included on OS level in the Semeru container images?
Is there any container image I could simply use for my evaluation, or do I need to adapt the following file somehow and see what I can get running?
https://github.com/ibmruntimes/InstantOnStartupGuide/blob/main/Containerfiles/Containerfile.ubuntu22.unprivileged
The text was updated successfully, but these errors were encountered: