Trunk builds failing

[mattgodbolt]

Hi @gautam1168 - the (forked from here) CE daily build has been failing for a while with:

2023-06-30T00:34:42.7650998Z + python ./tools/dev/gm.py x64.release
2023-06-30T00:34:42.9233547Z ERROR at //BUILD.gn:2167:5: Unable to load "/root/v8/third_party/abseil-cpp/BUILD.gn".
2023-06-30T00:34:42.9234012Z     "//third_party/abseil-cpp:absl_define_config",
2023-06-30T00:34:42.9234356Z     ^--------------------------------------------

e..g https://github.com/compiler-explorer/compiler-workflows/actions/runs/5417971736/job/14672787198 and all around that time.

Any ideas? Thanks!

[gautam1168]

I will investigate today and let you know

[gautam1168]

I have some good news and some bad news.

• Good news: I can build it successfully using the container made from my repository
• Bad news: I did that accidentally. I meant to do the build from the latest container of this repository and used mine instead. So, now I'm starting a new build to do it starting with the dockerfile here. Since I am able to do it using the other repo I suppose whatever the issue is, should be fixable. I just need to figure it out. I'll update once I know what the fix is.

[gautam1168]

Ah actually I used the correct repository: https://github.com/compiler-explorer/d8-builder

So I made a fresh container from the fork that ce uses and I was able to do the build successfully. hmm.. this is strange. So the build is failing in the CI only it would seem. I did the build by starting a bash session inside the container and running the build manually. Let me try to trigger the build using the script build.sh as it would be done in the automated system. Maybe that will reproduce it.

[gautam1168]

@mattgodbolt will it be possible to recreate that docker image? I think that will fix the problem. If it does, then I'd like to modify the build script to not do a git pull which requires it to merge changes (which is the problem here I think) and instead delete local main branch and get the remote one. I can't reproduce this issue locally (inside docker desktop) so far, but Ill keep trying while I wait for your reply.

[mattgodbolt]

@gautam1168 I'll try and work out how to recreate! thanks!

[gautam1168]

If it works let me know and I'll open a PR to not do a git pull. If it doesn't work I will still need to reproduce this somehow.

[mattgodbolt]

One way to fix both the docker-recreate issue and the git pull is to put in the docker image itself the SHA of the version to git checkout -- so...

git fetch && git checkout KNOWNGOODSHA

then if we change the SHA it forces the image to rebuild too. Do you know such a checkout that would be appropriate?

[gautam1168]

Well we don't know the SHA before we do the fetch. Right now when the trunk build happens we do the following:

git checkout main
git fetch
git pull

And I was thinking we could change it to

git checkout -b temp
git branch -D main
git fetch
git checkout main

This would remove the need for a pull. But I would like to confirm that this pull is the cause of the build failures before this change is made.

[mattgodbolt]

Running now: https://github.com/compiler-explorer/compiler-workflows/actions/runs/5664634020 (but I am now afk to go to bed..)

[gautam1168]

So it worked. Although in the logs I see there is still a pull. I am not really certain why the old container was failing but the pull is the only thing I can think of. Should I make a PR so that there won't be a git pull involved as I mentioned before?

Well actually I can think of one more thing. The gclient sync command only runs when the container is built and that command updates the build tools. The failure was happening in gm.py though and that is not in the depot_tools. Its in the v8 repository itself. Nevertheless I can add a call to gclient sync each time a build is done as well.

[mattgodbolt]

So it worked. Although in the logs I see there is still a pull. I am not really certain why the old container was failing but the pull is the only thing I can think of. Should I make a PR so that there won't be a git pull involved as I mentioned before?

I don't really understand I'll be honest: if the build ran OK why change?

gclient in the container

right - the container step is more an "acceleration" I thought? I don't really mind either way: I'm not hugely invested in this other than having it working :)

Thanks!

[gautam1168]

I don't really understand I'll be honest: if the build ran OK why change?

I was thinking about how to prevent it from breaking again in the future.

right - the container step is more an "acceleration" I thought? I don't really mind either way: I'm not hugely invested in this other than having it working :)

So the container has this line:

RUN cd ~ && fetch v8 && cd ~/v8 && gclient sync \
        && sed -i 's/${dev_list} snapcraft/${dev_list}/g' build/install-build-deps.sh \
        && cd ~/v8/build && ./install-build-deps.sh && git reset HEAD --hard \
        && chown -R root:root /root/v8

Here the gclient sync performs an update on the build tools. I think this could possibly have been the reason why rebuilding the container made it work.

If you are comfortable with this for now we could just leave it as it is now. If it breaks again in the future I can make these changes to make it more stable.

[mattgodbolt]

Thanks @gautam1168 - I apprecaiate the context. For now I don't have a strong opinion either way. Much appreciated :)