Anyone run into the linux docker base build failing at step 30 RUN gclient sync with cipd failing with invalid cross-device link?

[josh-hemphill]

Not sure yet if this warrants an issue here or in the chromium-dev groups. I assume others can run and build the docker containers just fine?
I think I've eliminated it being my host machine, I've run it on 2 different physical machines, a vm, and tried with 2 different container backends.

I actually also run into the RUN fetch v8 failing with tar permissions, but I can set the tar option --no-same-owner and that goes along fine but fails at gclient sync. I can also adjust the file to run as a non-root user which solves the permissions issue and everything runs up until the exact same error.

I was going to work on a build for linux arm64, but this happens in the base build whether I change things or not.

The exact error is:

STEP 30/71: RUN gclient sync
Updating depot_tools...

________ running 'cipd ensure -log-level error -root /google -ensure-file /tmp/tmpb5jmmqu3.ensure' in '.'
[P730 08:39:52.369 client.go:1901 E] [cleanup] Failed to remove infra/3pp/tools/ninja/linux-amd64 in "v8/third_party/ninja": removing the deployed package directory: rename /google/.cipd/pkgs/1 /google/.cipd/pkgs/B9dhvBqFSHhs: invalid cross-device link
Errors:
  failed to remove infra/3pp/tools/ninja/linux-amd64 in "v8/third_party/ninja": removing the deployed package directory: rename /google/.cipd/pkgs/1 /google/.cipd/pkgs/B9dhvBqFSHhs: invalid cross-device link
Error: Command 'cipd ensure -log-level error -root /google -ensure-file /tmp/tmpb5jmmqu3.ensure' returned non-zero exit status 1
[P730 08:39:52.369 client.go:1901 E] [cleanup] Failed to remove infra/3pp/tools/ninja/linux-amd64 in "v8/third_party/ninja": removing the deployed package directory: rename /google/.cipd/pkgs/1 /google/.cipd/pkgs/B9dhvBqFSHhs: invalid cross-device link
Errors:
  failed to remove infra/3pp/tools/ninja/linux-amd64 in "v8/third_party/ninja": removing the deployed package directory: rename /google/.cipd/pkgs/1 /google/.cipd/pkgs/B9dhvBqFSHhs: invalid cross-device link

Error: building at STEP "RUN gclient sync": while running runtime: exit status 1

[caoccao]

It's been broken in docker for months. I think that's a docker issue. This post might work, but I haven't tested that because I always use physical box.

[josh-hemphill]

So I think I'm going to still open a bug with cipd in hopes someday there's a way for rename() to handle XDEV errors gracefully while maintaining atomicity. (looks like there's a flag option for the function for creating an intermediate file on overlay filesystems, which might be what I'm looking for, so there's hope)
In the meantime I'll look at writing some scripts and keep them as concise and straightforward as possible.

[caoccao]

Thank you for the update.

The challenge is the Linux and Android base images share the same layers till fetch v8 and install-build-deps.sh. If all these commands are packaged in one script, the cache will not be reused. Why did I design it that way? Well, because I'm behind GFW. My VPN service charges me by data volume. 20GB per image is not a good thing for my wallet.

So, if someone could take care of the docker image build and push, I would be fine with a refactor of the docker files.

[josh-hemphill]

I just realized I could just copy>delete-origonal>move-back the whole /google directory so it's owned by the layer before running gclient sync.
I hate doing it, but in this case a bunch of extra time to the build isn't as bad as the download.

[caoccao]

Actually, you don't need to call gclient sync inside the docker if you performs gclient sync locally and copy everything over. But, that implies others are not able to build the images as you do.

[caoccao]

By the way, there's another issue #171 on redesigning the dockerfiles. I posted my suggestions some of which touch the Android image in that issue. You are welcome joining the discussion. Cheers 😄