Upload docker image to dockerhub

[sornian]

https://hub.docker.com/u/sjtucaocao

Images in docker hub is not the new version

I try to build base.docker but our country can't visit Google any website

Could you please upload the new image to docker hub

Please

[caoccao]

Thank you for raising this issue. I have been trying every week, however I have failed to do so successfully over the past 2 weeks because of the VPN inability 😭.

[sornian]

Most developers only use javet or edit the javet source code without directly modifying the source code of V8
use the v8 compiled file is enough
could we upload the v8 compile file to github?

[caoccao]

I don't think that is a good idea. Those binary files could reach tens of GBs so that cloning this project would be impossible.

[amithgeorge]

@caoccao we could setup the github actions, such that any tagged commit gets its docker image built and push to dockerhub. I could do a PR for that if needed.

[caoccao]

we could setup the github actions, such that any tagged commit gets its docker image built and push to dockerhub. I could do a PR for that if needed.

Here are the challenges for your reference.

1. Building the base image takes 1-2 hours. That would breach Github action execution limit (1 hour) and be killed.
2. Uploading the base image to docker hub takes another 1-2 hours as the image size is 10-15GB.

If you could resolve these challenges, I would be happy to merge your PR.

By the way, I'm using an VPN to get out of China mainland. That VPN charges me by the data volume. It would take 20+GB per base image. That's a heavy financial burden to me.

[amithgeorge]

Ok. I may have misunderstood something. What is the purpose of the uploaded docker image? From the dockerfile in the repo, it looks like the image is used to build the Javet binaries. Is the build image primarily for people wanting to contribute to Javet development? And not meant for users of Javet?

[amithgeorge]

Keeping aside the question of why is the image needed, we can tackle the concerns raised,

Building the base image takes 1-2 hours. That would breach Github action execution limit (1 hour) and be killed.

From the documentation for limits on actions, https://docs.github.com/en/actions/learn-github-actions/usage-limits-billing-and-administration#usage-limits

Job execution time - Each job in a workflow can run for up to 6 hours of execution time. If a job reaches this limit, the job is terminated and fails to complete.

Maybe the 1 hour limit you mentioned was an older limit? If I have missed something, please correct me.

Uploading the base image to docker hub takes another 1-2 hours as the image size is 10-15GB.

Do we need the source code to be present in the final built image? Docker has the concept of multi-stage builds, where we use one image to build the compiled files and then copy the compiled files to another image. The resultant final image is much smaller. As an example, using multi stage builds, a user was able to reduce the size of the V8 image from 5GB to 0.47GB. Reference

We can apply the same concept to building Node (though I am not sure why are we building Node instead of installing it). To speed up the overall build, we can choose to build the V8 image independently from the Node image and refer both in the multi-stage build for Javet base. It won't be trivial to do, but if you could help me understand what is happening and what is actually needed (ie only compiled binaries and no source), then I can attempt something.

[caoccao]

Thank you for the help.

Ok. I may have misunderstood something. What is the purpose of the uploaded docker image? From the dockerfile in the repo, it looks like the image is used to build the Javet binaries. Is the build image primarily for people wanting to contribute to Javet development? And not meant for users of Javet?

It's for building and testing every commit.

I think I was perceived by some document about the 1 hour limit. Thank you for correcting me.

Regarding shrinking the image, it's tricky because Javet references the source code directly as well as the libraries. That implies the whole build environment, source code and .obj files are supposed to be in the image. You may take a try and find out how much you may drop. The largest image layer is 5+GB so that the connection to docker hub drops so frequently. Especially, as I'm behind the VPN, I haven't pushed such image successfully over the past few months. I doubt how stable the connection between github and docker hub.

[amithgeorge]

Reading through the Dockerfile and build scripts for linux, it looks like for the V8 build of Javet it only needs the V8 source and compiled folders. For the Node build of Javet, does it only need the Node source and compiled folders, or does it need the V8 ones as well?

If they are independent, then we can split the images, reducing their size further. However, if the Node build requires the V8 bits as well, we can base the Node image off the V8 image. The idea is instead of one huge 15gb image, we might have three images of 5gb each (these numbers are assumptions) - a V8 image, a Node image based off the V8 image, and a Javet image based off the Node image. The V8 and Node image will change maybe once every 4-6 weeks as per their release cycles. The 3rd Javet image may change more frequently, though that might be much smaller in size. Just ideas in my head atm.

[caoccao]

You are right. The dockerfiles can be layered as the diagram below. You are welcome making changes to the dockerfiles and workflow to achieve that goal. Thank you.

[amithgeorge]

Nice.

I had some trouble setting up an Ubuntu VM on my Windows 10 + WSL2 desktop. Need to figure out the details. Once that is done, I should be able to actually build the existing docker images. Give me a few days to figure things out.

[caoccao]

Nice.

I had some trouble setting up an Ubuntu VM on my Windows 10 + WSL2 desktop. Need to figure out the details. Once that is done, I should be able to actually build the existing docker images. Give me a few days to figure things out.

That will be great. Thank you very much.

[sornian]

嘿嘿

[amithgeorge]

@caoccao In my excitement for solving a technical challenge, I think I missed out on understanding the purpose of the image. Let me state my current understanding and ask clarifying questions.

The latest image on https://hub.docker.com/r/sjtucaocao/javet/tags is tagged 1.1.5. This image is created from the dockerfile docker/linux-x86_64/base.Dockerfile. You mentioned in an earlier message that the purpose of the docker image is

"It's for building and testing every commit."

I can see in the Github workflow, that this image is used to test the code on every pull request, and to build the jar files and upload them as artifacts.

1. Is that the only place where this docker image is used? Will we (you or any contributor to Javet source) also be using the docker image for local development?
2. If we think about it, the base image only contains the V8 and Node source/binaries. Let's ignore the jar dependencies downloaded using gradle for now. If we consider only the V8 and Node dependencies, then the name of the docker image and it's tag should change. Sample image name and tag - javet-dev:v8-10.3.174.14_node-16.15.1. I feel tagging this way makes it more obvious what the image is meant for, as opposed to naming it javet:1.1.5. I am open to a different naming scheme.

3.1. I understand the desire to cache the dependency jar files using gradle in the base.Dockerfile. I would suggest achieving that purpose in a different way. Let's change the purpose of the docker/linux-x86_64/build.Dockerfile file. Currently as part of building the image, the artifact jar is created. Meaning the image itself has no value, except for performing a side effect. That's why we have this awkward step of running the image and copying the built artifacts.

3.2. Instead we strip the build.Dockerfile of almost all the commands, keeping it bare. We build the image and run the container. During the run, we mount the current source code directory at the path /Javet and provide a shell script to execute. The shell script does the Build JNI and Build Jar steps. The jar is written to the mounted path. Once the container has finished execution, the artifact jars are already available for uploading. The advantage of this approach is, along with mounting the source code, we can also mount the path where the gradle dependencies are stored. This lets us cache the gradle dependencies in the Github actions cache and reuse them across builds. If a developer ever uses this docker image locally, they can mount their local source code folder and the local gradle dependencies folder.

What do you think about the points I raised? Am I missing anything?

[sornian]

Ok. I may have misunderstood something. What is the purpose of the uploaded docker image? From the dockerfile in the repo, it looks like the image is used to build the Javet binaries. Is the build image primarily for people wanting to contribute to Javet development? And not meant for users of Javet?

maybe most users only use Javet products, but some customization needs should be taken into account. This requires this image to build Javet. Uploading to DockerHub is a quick way

[amithgeorge]

@caoccao An update. I was able to build the V8_Node image using a GitHub Action. It took a little over 3 hours!

The docker image (amithgeorge/javet-linux-dev:v8-10.6.194.14_node-18.10.0) contains the V8, Node and JVM. No Javet source is present, as I proposed in my earlier message.

The docker upload of the image from GitHub to DockerHub took around 7 mins.

Can you please use the image and test it? In the folder where you have the Javet source clone, please run the following. The gradle-home folder is used to cache the entire gradle home directory inside the container. I don't use gradle, so I am not sure which specific folders should we actually cache.

mkdir -p gradle-home
docker run -it --rm --name javet-dev-local \
   --volume $(pwd):/Javet --volume $(pwd)/gradle-home:/root/.gradle \
   amithgeorge/javet-linux-dev:v8-10.6.194.14_node-18.10.0 \
   bash

In the resulting shell, execute the following manually,

cd /Javet/cpp && \
  sh ./build-linux.sh -DV8_DIR=/google/v8 && \
  sh ./build-linux.sh -DNODE_DIR=/node && \
  cd ../ && \
    touch src/main/resources/libjavet-v8* && \
    gradle build test --rerun-tasks --debug && \
    touch src/main/resources/libjavet-node* && \
    gradle test --rerun-tasks --debug

exit

In your local machine, the javet jar should be present at ./build/libs/

The GitHub runner has a hardware limitations to keep in mind. I don't think we will hit them anytime soon.
From https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources

Hardware specification for Windows and Linux virtual machines:
2-core CPU (x86_64)
7 GB of RAM
14 GB of SSD space

If you can confirm the docker image works, and is suitable for use, then I will continue with next steps.

---

Example workflow - https://github.com/amithgeorge/v8-node-source-binaries/blob/d71b4ea03f640587a3ed07a951fabcc3b831a218/.github/workflows/linux_build_dev.yml

Result of the action - https://github.com/amithgeorge/v8-node-source-binaries/actions/runs/3298958302

The docker image - https://hub.docker.com/layers/amithgeorge/javet-linux-dev/v8-10.6.194.14_node-18.10.0/images/sha256-42332c776174da067d0dd03dc4b97978a0c757ab44a1ac14696d53a5fad18ff7

[caoccao]

Thank @amithgeorge for the great work. I'll check that out later.

Here are some tips for your reference.

• Gradle cache cannot be easily mounted and reused. It has to be generated inside the docker build. That's why gradle assemble is called during the base image building phase. Gradle doesn't calculate universal consistent path for the cache, which means the cache path varies per user, per instance. So, I don't think your design will leverage the gradle cache.
• The current design is to minimize the time on building and testing the final binaries. Which means the work is done in the base image as much as possible.
• The Linux and Android base images share many steps till install-build-deps.sh to save time on building, pushing, pulling. Hope the new design keeps that.

[sornian]

Thank @amithgeorge for the great work。Can you upload Android and Windows image

[amithgeorge]

@sornian this is still a work in progress. The linux image is currently uploaded to my person DockerHub repo. Ideally, these images should be hosted on the Javet DockerHub. So, I would strongly advise against relying on the images I am uploading. These are mainly to test whether I am on the right track. I do not guarantee these images will remain for more than a few days.

I have never worked on Android/ARM or Windows docker images before. So I am not even sure whether I would be able to contribute towards that. My idea was to setup something for Linux, bring it under the Javet org/brand. Then @caoccao or someone with appropriate experience can rely on that to similarly implement the infra for Android and Windows.

From @caoccao's earlier message, it sounds like there is some overlap between Linux and Android images, so I might take a stab at Android as well later.

[caoccao]

I tried it out today. Thank @amithgeorge for the excellent work!

Here are few suggestions.

• It's better to call gradle assemble in the base image to prepare the gradle cache so that the actual build time will be reduced. The typical use case is Javet developers repeatedly build and test the binaries for quick verification. Keeping that loop as fast as possible should be considered seriously and the current design fulfills that.
• The actual build on top of the base image currently is inside another dockerfile. I'd like to keep this design for the simplicity. I think calling a bunch of commands is not friendly to the developers who just want to change a few lines of code to see if that works. Also, as the file systems are quite different between Windows and Linux, that dockerfile fills that gap so that building the Linux binaries on Windows reuses exactly the same one-line command.
• There are some issues Anyone run into the linux docker base build failing at step 30 `RUN gclient sync` with cipd failing with `invalid cross-device link`? #196 with the Android base image. You are welcome joining the discussion. I believe the solution would be quite different from the current approach you take for the Linux build. But, as long as they are separate images built by Github actions, I'm fine with no overlaps at all.

Please feel free to create a new branch with the changes and I'll review it whenever you want me to. Thank you again for you great help.

[josh-hemphill]

I've also been working on getting the docker builds working, and did some experimentation breaking it into a multi-stage build.
I ran into issues with gclient sync since it's use of rename() fails when trying to move files across layer boundaries, and until the bug I submitted is processed, I'm doing a whole copy-delete-moveback to bring all the code into the same layer as gclient, which is kind of awful.

I'm ultimately trying to get arm64 working so it can be used on some of the raspis some of my friends are running, so I've been working with combinations of --platform= and cross-compilation flags.
It seems like with the multi-stage build, I can just add a few args and have the android actually just pull from a stage of the linux build, or combine then into a single file.
I'm just having trouble deciding how to do cpu architectures, since docker emulation works fine but will be a lot slower to build any non-native architectures, so I'm leaning towards custom build args to parse out what additional cross-compilers need to be installed so you can build for all your desired platforms, sharing the source code, but that's going to get very complex, so I'm not sure what the preference is.

[josh-hemphill]

There might be some confusion, I ran into the RUN gclient sync using the linux dockerfile, not the android one.
It seemed to be a docker/container/overlayfs issue, nothing to do with the build target.

[caoccao]

There might be some confusion, I ran into the RUN gclient sync using the linux dockerfile, not the android one. It seemed to be a docker/container/overlayfs issue, nothing to do with the build target.

gclient sync behaves abnormally if the target_os is a non-native one, I think. That might contribute to the docker/container/overlayfs issue. I think a shell script with all these commands might work. Would you mind taking a try?

[josh-hemphill]

I don't think target_os has anything to do with the error I was having, it's never set in the linux dockerfile. I ran and observed the error in the unedited linux/base.Dockerfile with both overlayfs fuse-overlayfs and overlay2 storage divers, native and non-native filesystems, and inside and outside docker emulation; all resulted in an identical error.

[caoccao]

I don't think target_os has anything to do with the error I was having, it's never set in the linux dockerfile. I ran and observed the error in the unedited linux/base.Dockerfile with both overlayfs fuse-overlayfs and vfs storage divers, native and non-native filesystems, and inside and outside docker emulation; all resulted in an identical error.

That's weird because it works in my environment (Windows 10 + WSL 2 + Docker + Ubuntu 20.04). May I know your environment?

[josh-hemphill]

I tried on both my Ubuntu 22.04 systems, but also inside a podman container VM which uses fedora-coreos-36.20221014.2.0.

So it is the kernel that's kind of responsible, but from what I could tell, the rename() kernel function responsible hasn't changed its behavior between kernel versions, so it would have to be on some kind of wrapping library that I guess used to hide/ignore XDEV errors, but now propagates them so things know when they're breaching filesystem boundaries and loosing atomicity.

After talking to someone from the chromium mailing group, it sounds like the libs that gclient uses rely on atomicity when moving files, so this is actually better in a way since it will ensure we don't end up with gclient silently clobbering files or other undefined beaviour.

[josh-hemphill]

I found another upside to the multi-stag approach is it seems to multi-thread very nicely with the --jobs=0 option set. (though I've been setting to $(($( getconf _NPROCESSORS_ONLN ) - 1 )) so it will max out at 1 less than the number of physical cores available.)

[caoccao]

I wonder if mounting a physical volume as /google_tmp would get rid of that invalid cross-device link issue, and later copy /google_tmp to /google. Would @josh-hemphill take a try?

[josh-hemphill]

It would get rid of the error, but that's just because you're copying the data, which makes it owned by that layer.
So I'm thinking a good solution is to have previous stages use another directory, then copy into the /google directory in the same layer as gclient sync and just make sure that only the /google directory gets persisted to the next stage.
Whether or not to rm the old directory I think is up in the air. In my experience it does add some time to the build, but not too much. If I don't rm the old directory that stage will have a much larger footprint, but it shouldn't persist into any final image, so I'm not sure whether to do the rm or not.

[caoccao]

If I don't rm the old directory that stage will have a much larger footprint, but it shouldn't persist into any final image, so I'm not sure whether to do the rm or not.

Right, that's also one of my concerns. The image size might be nearly doubled if that's not taken well care of. Let the test tell us if that's a valid concern.

[amithgeorge]

@caocao,

I was trying to reduce the time it takes to build the image on GH and reduce the overall size of the final image.

1. I am doing a shallow clone of V8 and Node.js. I modified the gclient instructions accordingly. Also combined the various apt-get commands into one RUN layer, to ensure that cleaning the apt cache actually results in a size reduction. As it stands, the on disk image size has reduced by almost 5GB. The compressed size on DockerHub has reduced by almost 3GB (5.75GB vs 2.83GB).

amithgeorge/javet-linux-dev   v8-10.6.194.14_node-18.10.0_shallow   00fea6d1b64d   15 hours ago   6.77GB
amithgeorge/javet-linux-dev   v8-10.6.194.14_node-18.10.0           065d339cbfab   2 days ago     11.7GB

2. I split the build into multiple dockerfiles, to create independent images

• base-jvm : has the updated Ubuntu, JVM and build tools
• base-v8 : is based off base-jvm and has V8 source and compiled binaries
• base-node : is based off base-jvm and has Node.js source and compiled binaries
• v8_node: is based off base-jvm and has the relevant Node.js and V8 folder copied over

With this setup, I can build the v8 image in parallel to the Node.js image. Doing a full run of the workflow, it takes only 1h45m to build all the images. Earlier, when done sequentially, it took over 3h.

3. If we really need to cache the gradle dependencies, then it is trivial to build an image based off v8_node and download the dependencies in it. Will do so. Just to make it clear, the 11.7GB image and the 6.77GB image both don't have gradle deps in them. So, the size reduction is comparable.

4. The current workflow is setup such that we manually trigger the build. I am not sure how often you bump up the Node.js version and V8 version. However in the current setup, we can bump up the version independently, and when running the workflow, chose to only build the updated dependency image and the final images.

5. Let me know if you have any concerns with the approach I am taking, with splitting the docker images. My main reason to do so is the speed up the overall build and reuse any existing images where possible.

6. I will add another job to the workflow to create the image as you suggested, ie including the gradle dependencies. Please wait a bit for it. I will request you to test it again once I have built that.

Links
Workflow UI - https://github.com/amithgeorge/v8-node-source-binaries/actions/runs/3308169436

Worflow File (inputs to skip building a certain image) - https://github.com/amithgeorge/v8-node-source-binaries/blob/0621ee4e638bc11e7b31372580d3acc8ed2b67db/.github/workflows/linux_build_dev_parallel.yml#L8

Script to shallow clone and build V8 - https://github.com/amithgeorge/v8-node-source-binaries/blob/0621ee4e638bc11e7b31372580d3acc8ed2b67db/scripts/shell/fetch_v8_source

Final image (no gradle deps yet, will add that later) - https://hub.docker.com/layers/amithgeorge/javet-linux-dev/v8-10.6.194.14_node-18.10.0_shallow/images/sha256-c02d7c80bd5e50274b00e2e7789e03906ce668d422e94ad3cd4b34c43f7fada1?context=explore

[josh-hemphill]

I did some experiments with emulated arm64 builds and it seems that it introduces enough odd behavior into Node.js's c and c++ compiles that I'm going to have to do down the path of installing the cross-compilers for arm builds when specified.
The only issue of doing it that way is more layers will be invalidated (hopefully still downstream of the main chunk of the downloads) whenever you want to change which targets you generate.

It should technically be possible to have a single linux dockerfile build anything in this matrix

	x86_64	ia32	arm64	arm
android	✅	✅	✅	✅
linux	✅	✅	✅	✅

The only issue becomes (might be worth referencing the build-nodejs-for-android docker container and its use of the android-gcc-toolchain to do a few more):

	android_x86_64	android_ia32	android_arm64	android_arm	linux_x86_64	linux_ia32	linux_arm64	linux_arm
v8	✅	✅	✅	✅	✅	✅	✅	✅
NodeJS	?	?	?	?	✅	✅	~	~

Which I'm not sure an argument parser will be able to help with since the only way to skip a stage is with an environment variable that's set with ENV or a raw ARG, so best case scenario I think there will be additional build args required when building, though having defaults set for only building linux_x86_64 should make it so the only people that need it to do releases or are doing something platform-specific will need to use the additional args

[caoccao]

Thank @amithgeorge for the wonderful update.

1. Your mod looks good to me as long as the build passes.
2. The dockerfile hierarchy matches the diagram I posted. 1'45'' looks awesome!
3. Yes, it's better to have a new image on top of those images with the gradle dependencies.
4. The version increase only happens once per month. I think manually triggering the build is the right approach.
5. I agree with your approach.
6. I look forward to that.

Cheers!

[caoccao]

Thank @josh-hemphill for the update.

It should technically be possible to have a single linux dockerfile build anything in this matrix

I prefer them being separate images between the Android and Linux. I hold concerns on a mega image with both mainly because of the following.

• The image size would be nearly doubled.
• The build would be sequential with longer time.

The only issue becomes (might be worth referencing the build-nodejs-for-android docker container and its use of the android-gcc-toolchain to do a few more)

As Node.js doesn't officially support Android, I don't want to support it on Android. Regarding Linux arm/arm64, someone else did that successfully.

Which I'm not sure an argument parser will be able to help with since the only way to skip a stage is with an environment variable that's set with ENV or a raw ARG, so best case scenario I think there will be additional build args required when building, though having defaults set for only building linux_x86_64 should make it so the only people that need it to do releases or are doing something platform-specific will need to use the additional args

That is something I have been holding myself not to do. I'd rather keep the build system:

• as straight forward as possible
• decoupled from other OS/CPU Arch
• easy to be maintained/upgraded

Why? The build system of J2V8 left me with some negative experience. You are welcome taking a look at it as it is very close to what you described. I'm here against code reuse in such system because the external dependencies could easily break the assumptions relied on by the reusable modules and that would result in the collapse of the whole build system.

[josh-hemphill]

Thanks for the clarification @caoccao .

Regarding:

• The image size would be nearly doubled.
• The build would be sequential with longer time.

I wasn't suggesting combining the images; I'm writing a multi-stage dockerfile just so that the shared layers at the beginning are explicitly referenced and the same command/file can be used to build either (theoretically you could do both) just by changing the arguments you pass for which targets you want to build.

As for keeping it as simple as possible, adding something like

docker build --jobs=0 ./docker/linux/base.Dockerfile .

docker build --jobs=0 --build-arg TARGETS=x86_64,ia32,arm64,arm ./docker/linux/base.Dockerfile .

to get x64.release and x64.release,x86.release,arm64.release,arm.release makes a lot more sense; instead of

docker build --jobs=0 ./docker/linux-x86_64/base.Dockerfile .
docker build --jobs=0 ./docker/linux-ia32/base.Dockerfile .
docker build --jobs=0 ./docker/linux-arm64/base.Dockerfile .
docker build --jobs=0 ./docker/linux-arm/base.Dockerfile .

when more than 80% of the files are the same.

Most people aren't going to build the base.Dockerfile themselves, and the one pushed to docker hub can have all (x86_64,ia32,arm64,arm) so there only needs to be one build.Dockerfile as well, which won't need arguments or anything. In fact, by doing it that way the build.Dockerfile can reference the base.Dockerfile by COPY --from=sjtucaocao/javet ... ... so if people want to build containers with different OSs or ARCHs, it already has the build sources so it could "just work".

[caoccao]

Thank @josh-hemphill for the detailed explanation. I agree with the approach you proposed. Let's see if that could be coordinated with the redesign that @amithgeorge is working on.

Would you mind joining the discord to have a group discussion in real-time? Thank you.

[josh-hemphill]

I joined the discord, but I've got to get some get some sleep, so I'll have to talk tomorrow.
But in the meantime I wanted to share an example of what the multi-stage builds looks like:
https://gist.github.com/josh-hemphill/0813c82f3eb6550b27dd1a1279fed1e8
Still have some errors, so it's not a working example, but it should show what the idea is.

[caoccao]

But in the meantime I wanted to share an example of what the multi-stage builds looks like:
https://gist.github.com/josh-hemphill/0813c82f3eb6550b27dd1a1279fed1e8
Still have some errors, so it's not a working example, but it should show what the idea is.

@josh-hemphill In general, that looks fine. Here are few tips for your reference.

• Please make sure it works on both Linux and Windows (with WSL 2).
• It's better to separate the Android image to reduce the overall image size.
• Proper comments may help people better understand those steps.
• The Node.js and V8 version might be written somewhere else so that the dockerfile could remain unchanged for quite a while in the future.

Thank you.

[caoccao]

@amithgeorge @josh-hemphill I just merged #216 to main. Please let me know if it matches the multi-staged build that you think.