dart-sdk linux redistributable has incorrect permissions for some directories

[asottile]

I'm using docker to reproduce as otherwise umask takes effect while extracting:

root@dbe466e80cf1:/tmp/y# curl -o dart.zip https://storage.googleapis.com/dart-archive/channels/stable/release/2.13.4/sdk/dartsdk-linux-x64-release.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  181M  100  181M    0     0  64.2M      0  0:00:02  0:00:02 --:--:-- 64.2M
root@dbe466e80cf1:/tmp/y# unzip -q dart.zip
root@dbe466e80cf1:/tmp/y# ls -ald dart-sdk/
drwx------ 5 root root 4096 Jun 23 11:40 dart-sdk/

note in this case the mode of the extracted directory is 700 but I expect it to be 755

there are other directories in the archive which suffer from the same problem

I'm currently working around this by using chmod -R og+rX

[athomas]

Yes, this is wrong. We want drwxr-xr-x (755) on directories, -rwxr-xr-x (755) on executables and -rw-r--r-- (644) on other files.

[asottile]

I suspect this is due to mkdtemp somewhere, I can probably fix it if I have a hint where the build tooling is and how to run it

[athomas]

Creating your own build is documented here: https://github.com/dart-lang/sdk/wiki/Building

Not sure it's that straightforward, the dart-sdk produced by the build already has the wrong permissions. Then a python script zips up the mess (some files have permissions coming from git, some only have owner permissions probably those created by the build). Perhaps we need to do some post-processing similar to what chromium seems to be doing.

The SDK build is tied together by sdk/BUILD.gn but it's not obvious to me what exactly creates the dart-sdk directory (I suspect it's created as a side effect of copying other files in that directory).

[asottile]

I tried to build this, swapped to death (seemed to have 3k+ processes trying to build). so then I downloaded more ram and even 24GB was not enough to build so I'm kinda out of ideas on what to start trying here

I did notice a lot of mkdtemp throughout the build tooling and that'll create directories which are 0o700 instead of 0o755 which is probably the problem, but I don't really want to guess at where to fix it and I can't seem to run the build myself 😭

[athomas]

@asottile I'd be interested to know how you invoked build.py. If nothing else, I could fix the build script improve the documentation. It sounds to me like the build ran with a high -j but without the internal distributed compiler and then tried to do all the work with a high parallelism on a single machine. That shouldn't happen so we've got a bug somewhere.

[asottile]

this is the command I ran:

./tools/build.py --no-goma -j4 --mode debug --arch x64 create_sdk

[athomas]

this is the command I ran:

./tools/build.py --no-goma -j4 --mode debug --arch x64 create_sdk

Thanks, I haven't been able to reproduce this on my machines (nothing close to 1000s of processes and on my workstation memory usage was below 10GB the whole time). You may want to try --mode release because that's how we normally build the SDK (debug is more used to debug the VM and is probably slow on the create_sdk target).

$ ./tools/build.py --no-goma -j4 --mode release --arch x64 create_sdk
Done. Made 320 targets from 89 files in 266ms
ninja -C out/ReleaseX64 create_sdk
ninja: Entering directory `out/ReleaseX64'
[3231/3231] STAMP obj/create_sdk.stamp
The build took 479.772 seconds

[asottile]

I also tried --release as well -- can seek around in the video but this was the last command I tried: https://www.twitch.tv/videos/1167569850?t=2h57m0s

[athomas]

@asottile as far as I can see you didn't really follow the Getting the source instructions. After having depot_tools in the path, these is what gives you a working checkout:

mkdir dart-sdk
cd dart-sdk
# On Windows, this needs to be run in a shell with Administrator rights.
fetch dart

You might be able to fix-up your checkout by running gclient sync in it (but it might require some additional configuration that fetch dart would've done for you). gclient sync is what provisions the required tooling (e.g. the buildtools directory).

PS.: I think you engineered yourself some sort of fork-bomb by putting chromium's gn helper script in the place where the gn binary is expected. Which would explain the many processes.