Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Building Dart SDK fails at fetch dart on Apple Silicon / DTK / macOS 11 #43488

Closed
alexmarkley opened this issue Sep 19, 2020 · 11 comments
Closed

Building Dart SDK fails at fetch dart on Apple Silicon / DTK / macOS 11 #43488

alexmarkley opened this issue Sep 19, 2020 · 11 comments
Labels
area-vm Use area-vm for VM related issues, including code coverage, FFI, and the AOT and JIT backends. os-osx

Comments

@alexmarkley
Copy link

This is almost certainly not an issue in the Dart SDK proper, but an upstream tooling issue. That said I don't have a better place to report this, so here we go...

Steps to Reproduce
Follow these instruction on building the Dart SDK: https://github.com/dart-lang/sdk/wiki/Building

  • Install depot_tools
  • do fetch dart

Observed Behavior
fetch dart cannot download the Dart SDK source code as per the documentation.

Expected Behavior
I would like to be able to download the Dart SDK source code.

Runtime Environment

DTK:dart-sdk alex$ uname -a
Darwin DTK.mylocal 20.1.0 Darwin Kernel Version 20.1.0: Sat Sep 12 20:05:59 PDT 2020; root:xnu-7195.40.84.171.4~1/RELEASE_ARM64_T8020 arm64
DTK:dart-sdk alex$ which python
/usr/bin/python
DTK:dart-sdk alex$ python --version
Python 2.7.16
DTK:dart-sdk alex$

Notes on Python 2
As per the comment at https://github.com/dart-lang/sdk/wiki/Building#python-2 I did also try installing virtualenv and creating a python2 virtualenv prior to fetching the SDK. This had no discernible effect, since the system default python is already python 2.7.

Why am I trying to do this?
I realize that Dart is available to download for many architectures without compiling from source. I want a native (preferably "universal") toolchain for my Flutter iOS & macOS development purposes, with the ultimate goal of targeting universal macOS desktop binaries for my Flutter app. I am interested in helping in the porting effort by reporting issues I encounter, testing experimental builds, and even writing some patches if I can.

Log

DTK:Build alex$ which fetch
/Users/alex/Build/depot_tools/fetch
DTK:Build alex$ mkdir dart-sdk
DTK:Build alex$ cd dart-sdk
DTK:dart-sdk alex$ fetch dart
Running: gclient root
WARNING: Your metrics.cfg file was invalid or nonexistent. A new one will be created.
Running: gclient config --spec 'solutions = [
  {
    "name": "sdk",
    "url": "https://dart.googlesource.com/sdk.git",
    "deps_file": "DEPS",
    "managed": False,
    "custom_deps": {},
  },
]
'
Running: gclient sync

________ running 'arch -arch arm64e -arch arm64 git -c core.deltaBaseCacheLimit=2g clone --no-checkout --progress https://dart.googlesource.com/sdk.git /Users/alex/Build/dart-sdk/_gclient_sdk_4o5a0hy_' in '/Users/alex/Build/dart-sdk'
Cloning into '/Users/alex/Build/dart-sdk/_gclient_sdk_4o5a0hy_'...
remote: Sending approximately 1.05 GiB ...
remote: Counting objects: 1475, done
remote: Finding sources: 100% (34/34)
remote: Total 1044242 (delta 798152), reused 1044231 (delta 798152)
Receiving objects: 100% (1044242/1044242), 1.04 GiB | 47.34 MiB/s, done.
Resolving deltas: 100% (798152/798152), done.
Syncing projects: 100% (100/100), done.                                                                   

________ running 'download_from_google_storage --no_auth --no_resume --bucket dart-dependencies --recursive --auto_platform --extract --directory sdk/third_party/firefox_jsshell' in '/Users/alex/Build/dart-sdk'
0> Failed to fetch file gs://dart-dependencies/6cc132344cae69172a69cca2ff503da84e5b9c6c for sdk/third_party/firefox_jsshell/mac/jsshell.tar.gz, skipping. [Err: [E2020-09-18T20:15:21.743658-04:00 49300 0 venv.go:975] Command (cwd=/tmp/vpython_bootstrap972716080/packages/virtualenv-15.1.0): [/usr/bin/python2.7 -B -E -s virtualenv.py --no-download /Users/alex/.vpython-root/f31a96]
Process output:
New python executable in /Users/alex/.vpython-root/f31a96/bin/python
ERROR: The executable /Users/alex/.vpython-root/f31a96/bin/python is not functioning
ERROR: It thinks sys.prefix is u'/private/tmp/vpython_bootstrap972716080/packages/virtualenv-15.1.0' (should be u'/Users/alex/.vpython-root/f31a96')
ERROR: virtualenv is not compatible with this system or executable

Environment:
AWS_CREDENTIAL_FILE=
BOTO_CONFIG=
HOME=/tmp/vpython_bootstrap972716080
LANG=en_US.UTF-8
LOGNAME=alex
PATH=/Users/alex/Build/depot_tools:/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Users/alex/Build/depot_tools:/Users/alex/Build/depot_tools
PIP_NO_BINARY=:none:
PIP_ONLY_BINARY=:all:
PIP_USE_WHEEL=1
PWD=/Users/alex/Build/dart-sdk
PYTHONDONTWRITEBYTECODE=1
PYTHONNOUSERSITE=1
SHELL=/bin/bash
SHLVL=1
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.hNHXObY3dW/Listeners
TEMP=/tmp
TERM=xterm-256color
TERM_PROGRAM=Apple_Terminal
TERM_PROGRAM_VERSION=439
TERM_SESSION_ID=DCF501EB-BE65-4402-BEB1-EBE63589ED69
TMP=/tmp
TMPDIR=/tmp
USER=alex
VERSIONER_PYTHON_VERSION=2.7
VIRTUAL_ENV=/Users/alex/.vpython-root/56f65e
XPC_FLAGS=0x0
XPC_SERVICE_NAME=0
__CFBundleIdentifier=com.apple.Terminal
__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
[E2020-09-18T20:15:21.748188-04:00 49300 0 annotate.go:241] original error: exit status 100
[E2020-09-18T20:15:21.748269-04:00 49300 0 annotate.go:241] 
[E2020-09-18T20:15:21.748282-04:00 49300 0 annotate.go:241] goroutine 1:
[E2020-09-18T20:15:21.748293-04:00 49300 0 annotate.go:241] #0 go.chromium.org/luci/vpython/venv/venv.go:615 - venv.(*Env).installVirtualEnv()
[E2020-09-18T20:15:21.748300-04:00 49300 0 annotate.go:241]   reason: failed to create VirtualEnv
[E2020-09-18T20:15:21.748313-04:00 49300 0 annotate.go:241] 
[E2020-09-18T20:15:21.748319-04:00 49300 0 annotate.go:241] #1 go.chromium.org/luci/vpython/venv/venv.go:529 - venv.(*Env).createLocked.func2()
[E2020-09-18T20:15:21.748334-04:00 49300 0 annotate.go:241]   reason: failed to install VirtualEnv
[E2020-09-18T20:15:21.748341-04:00 49300 0 annotate.go:241] 
[E2020-09-18T20:15:21.748349-04:00 49300 0 annotate.go:241] #2 go.chromium.org/luci/common/system/filesystem/tempdir.go:55 - filesystem.(*TempDir).With()
[E2020-09-18T20:15:21.748355-04:00 49300 0 annotate.go:241] #3 go.chromium.org/luci/vpython/venv/venv.go:103 - venv.withTempDir()
[E2020-09-18T20:15:21.748381-04:00 49300 0 annotate.go:241] #4 go.chromium.org/luci/vpython/venv/venv.go:515 - venv.(*Env).createLocked()
[E2020-09-18T20:15:21.748390-04:00 49300 0 annotate.go:241] #5 go.chromium.org/luci/vpython/venv/venv.go:272 - venv.(*Env).ensure.func1()
[E2020-09-18T20:15:21.748396-04:00 49300 0 annotate.go:241]   reason: failed to create new VirtualEnv
[E2020-09-18T20:15:21.748403-04:00 49300 0 annotate.go:241] 
[E2020-09-18T20:15:21.748412-04:00 49300 0 annotate.go:241] #6 go.chromium.org/luci/vpython/venv/venv.go:995 - venv.mustReleaseLock()
[E2020-09-18T20:15:21.748423-04:00 49300 0 annotate.go:241] #7 go.chromium.org/luci/vpython/venv/venv.go:258 - venv.(*Env).ensure()
[E2020-09-18T20:15:21.748430-04:00 49300 0 annotate.go:241] #8 go.chromium.org/luci/vpython/venv/venv.go:154 - venv.With()
[E2020-09-18T20:15:21.748443-04:00 49300 0 annotate.go:241]   reason: failed to create empty probe environment
[E2020-09-18T20:15:21.748451-04:00 49300 0 annotate.go:241] 
[E2020-09-18T20:15:21.748458-04:00 49300 0 annotate.go:241] #9 go.chromium.org/luci/vpython/run.go:62 - vpython.Run()
[E2020-09-18T20:15:21.748464-04:00 49300 0 annotate.go:241] #10 go.chromium.org/luci/vpython/application/application.go:320 - application.(*application).mainImpl()
[E2020-09-18T20:15:21.748477-04:00 49300 0 annotate.go:241] #11 go.chromium.org/luci/vpython/application/application.go:408 - application.(*Config).Main.func1()
[E2020-09-18T20:15:21.748484-04:00 49300 0 annotate.go:241] #12 go.chromium.org/luci/vpython/application/support.go:46 - application.run()
[E2020-09-18T20:15:21.748492-04:00 49300 0 annotate.go:241] #13 go.chromium.org/luci/vpython/application/application.go:407 - application.(*Config).Main()
[E2020-09-18T20:15:21.748499-04:00 49300 0 annotate.go:241] #14 vpython/main.go:106 - main.mainImpl()
[E2020-09-18T20:15:21.748514-04:00 49300 0 annotate.go:241] #15 vpython/main.go:112 - main.main()
[E2020-09-18T20:15:21.748520-04:00 49300 0 annotate.go:241] #16 runtime/proc.go:204 - runtime.main()
[E2020-09-18T20:15:21.748528-04:00 49300 0 annotate.go:241] #17 runtime/asm_amd64.s:1374 - runtime.goexit()
]
Failed to fetch file gs://dart-dependencies/6cc132344cae69172a69cca2ff503da84e5b9c6c for sdk/third_party/firefox_jsshell/mac/jsshell.tar.gz. [Err: [E2020-09-18T20:15:21.743658-04:00 49300 0 venv.go:975] Command (cwd=/tmp/vpython_bootstrap972716080/packages/virtualenv-15.1.0): [/usr/bin/python2.7 -B -E -s virtualenv.py --no-download /Users/alex/.vpython-root/f31a96]
Process output:
New python executable in /Users/alex/.vpython-root/f31a96/bin/python
ERROR: The executable /Users/alex/.vpython-root/f31a96/bin/python is not functioning
ERROR: It thinks sys.prefix is u'/private/tmp/vpython_bootstrap972716080/packages/virtualenv-15.1.0' (should be u'/Users/alex/.vpython-root/f31a96')
ERROR: virtualenv is not compatible with this system or executable

Environment:
AWS_CREDENTIAL_FILE=
BOTO_CONFIG=
HOME=/tmp/vpython_bootstrap972716080
LANG=en_US.UTF-8
LOGNAME=alex
PATH=/Users/alex/Build/depot_tools:/opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Users/alex/Build/depot_tools:/Users/alex/Build/depot_tools
PIP_NO_BINARY=:none:
PIP_ONLY_BINARY=:all:
PIP_USE_WHEEL=1
PWD=/Users/alex/Build/dart-sdk
PYTHONDONTWRITEBYTECODE=1
PYTHONNOUSERSITE=1
SHELL=/bin/bash
SHLVL=1
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.hNHXObY3dW/Listeners
TEMP=/tmp
TERM=xterm-256color
TERM_PROGRAM=Apple_Terminal
TERM_PROGRAM_VERSION=439
TERM_SESSION_ID=DCF501EB-BE65-4402-BEB1-EBE63589ED69
TMP=/tmp
TMPDIR=/tmp
USER=alex
VERSIONER_PYTHON_VERSION=2.7
VIRTUAL_ENV=/Users/alex/.vpython-root/56f65e
XPC_FLAGS=0x0
XPC_SERVICE_NAME=0
__CFBundleIdentifier=com.apple.Terminal
__CF_USER_TEXT_ENCODING=0x1F5:0x0:0x0
[E2020-09-18T20:15:21.748188-04:00 49300 0 annotate.go:241] original error: exit status 100
[E2020-09-18T20:15:21.748269-04:00 49300 0 annotate.go:241] 
[E2020-09-18T20:15:21.748282-04:00 49300 0 annotate.go:241] goroutine 1:
[E2020-09-18T20:15:21.748293-04:00 49300 0 annotate.go:241] #0 go.chromium.org/luci/vpython/venv/venv.go:615 - venv.(*Env).installVirtualEnv()
[E2020-09-18T20:15:21.748300-04:00 49300 0 annotate.go:241]   reason: failed to create VirtualEnv
[E2020-09-18T20:15:21.748313-04:00 49300 0 annotate.go:241] 
[E2020-09-18T20:15:21.748319-04:00 49300 0 annotate.go:241] #1 go.chromium.org/luci/vpython/venv/venv.go:529 - venv.(*Env).createLocked.func2()
[E2020-09-18T20:15:21.748334-04:00 49300 0 annotate.go:241]   reason: failed to install VirtualEnv
[E2020-09-18T20:15:21.748341-04:00 49300 0 annotate.go:241] 
[E2020-09-18T20:15:21.748349-04:00 49300 0 annotate.go:241] #2 go.chromium.org/luci/common/system/filesystem/tempdir.go:55 - filesystem.(*TempDir).With()
[E2020-09-18T20:15:21.748355-04:00 49300 0 annotate.go:241] #3 go.chromium.org/luci/vpython/venv/venv.go:103 - venv.withTempDir()
[E2020-09-18T20:15:21.748381-04:00 49300 0 annotate.go:241] #4 go.chromium.org/luci/vpython/venv/venv.go:515 - venv.(*Env).createLocked()
[E2020-09-18T20:15:21.748390-04:00 49300 0 annotate.go:241] #5 go.chromium.org/luci/vpython/venv/venv.go:272 - venv.(*Env).ensure.func1()
[E2020-09-18T20:15:21.748396-04:00 49300 0 annotate.go:241]   reason: failed to create new VirtualEnv
[E2020-09-18T20:15:21.748403-04:00 49300 0 annotate.go:241] 
[E2020-09-18T20:15:21.748412-04:00 49300 0 annotate.go:241] #6 go.chromium.org/luci/vpython/venv/venv.go:995 - venv.mustReleaseLock()
[E2020-09-18T20:15:21.748423-04:00 49300 0 annotate.go:241] #7 go.chromium.org/luci/vpython/venv/venv.go:258 - venv.(*Env).ensure()
[E2020-09-18T20:15:21.748430-04:00 49300 0 annotate.go:241] #8 go.chromium.org/luci/vpython/venv/venv.go:154 - venv.With()
[E2020-09-18T20:15:21.748443-04:00 49300 0 annotate.go:241]   reason: failed to create empty probe environment
[E2020-09-18T20:15:21.748451-04:00 49300 0 annotate.go:241] 
[E2020-09-18T20:15:21.748458-04:00 49300 0 annotate.go:241] #9 go.chromium.org/luci/vpython/run.go:62 - vpython.Run()
[E2020-09-18T20:15:21.748464-04:00 49300 0 annotate.go:241] #10 go.chromium.org/luci/vpython/application/application.go:320 - application.(*application).mainImpl()
[E2020-09-18T20:15:21.748477-04:00 49300 0 annotate.go:241] #11 go.chromium.org/luci/vpython/application/application.go:408 - application.(*Config).Main.func1()
[E2020-09-18T20:15:21.748484-04:00 49300 0 annotate.go:241] #12 go.chromium.org/luci/vpython/application/support.go:46 - application.run()
[E2020-09-18T20:15:21.748492-04:00 49300 0 annotate.go:241] #13 go.chromium.org/luci/vpython/application/application.go:407 - application.(*Config).Main()
[E2020-09-18T20:15:21.748499-04:00 49300 0 annotate.go:241] #14 vpython/main.go:106 - main.mainImpl()
[E2020-09-18T20:15:21.748514-04:00 49300 0 annotate.go:241] #15 vpython/main.go:112 - main.main()
[E2020-09-18T20:15:21.748520-04:00 49300 0 annotate.go:241] #16 runtime/proc.go:204 - runtime.main()
[E2020-09-18T20:15:21.748528-04:00 49300 0 annotate.go:241] #17 runtime/asm_amd64.s:1374 - runtime.goexit()
]
Downloading 1 files took 1.046263 second(s)
Error: Command 'download_from_google_storage --no_auth --no_resume --bucket dart-dependencies --recursive --auto_platform --extract --directory sdk/third_party/firefox_jsshell' returned non-zero exit status 1 in /Users/alex/Build/dart-sdk
Subprocess failed with return code 2.
DTK:dart-sdk alex$
This was referenced Sep 19, 2020
Closed
Closed
Closed
@lrhn lrhn added the os-osx label Sep 21, 2020
@alexmarkley alexmarkley mentioned this issue Sep 21, 2020
Closed
@alexmarkley
Copy link
Author

Bumping this because I'm still unable to download the Dart SDK source code. I'd love to contribute to testing/patching for Apple Silicon as a host, but I'm currently blocked on this issue.

@jmagman
Copy link

jmagman commented Sep 21, 2020 •
edited

\cc @a-siva @zanderso where's the best place to track this? If here, who can we poke to take a look?

As @alexmarkley mentioned it's surely not here, but maybe @a-siva knows how to work around it since he has it working? Or maybe the docs needs an update for ARM?

@jmagman
Copy link

jmagman commented Sep 21, 2020

https://bugs.chromium.org/p/chromium/issues/list ?

@alexmarkley
Copy link
Author

@jmagman Thank you, I'll take a look at that issue tracker and see what I can find. A cursory glance seems to show issues being tracked related to macOS 11 on both Intel and ARM, so hopefully there will be traction soon.

I'm also a little concerned because the Dart project explicitly calls out Python 2 as a requirement, and Python 2 is super EOL at this point.

@alexmarkley
Copy link
Author

alexmarkley commented Sep 21, 2020 •
edited

Seems like this might be the most immediately applicable upstream issue: https://bugs.chromium.org/p/chromium/issues/detail?id=1103275&q=component%3AInfra%3ESDK%20macos&can=2

In the issue it seems like partial workarounds are discussed, but I don't know enough about depot_tools to understand the discussion.

@mkustermann
Copy link
Member

Making Dart SDK buildable on Apple Sililcon is in-progress. @a-siva is working on this afaik.

@mkustermann mkustermann added the area-vm Use area-vm for VM related issues, including code coverage, FFI, and the AOT and JIT backends. label Sep 23, 2020
@alexmarkley
Copy link
Author

Some fixes have landed in MacPorts, and I was able to port install python27 which seems to have improved the above situation.

Now I'm hung up on what appears to be a CPU-bound process internal to depot_tools:

image

This thing has been running for a couple of days now and I'm not sure how to measure its progress. I'll report upstream as well...

@alexmarkley
Copy link
Author

More detailed notes upstream: https://bugs.chromium.org/p/chromium/issues/detail?id=1103275#c10

@Sunbreak
Copy link

Stuck here as well. What is 3pp build of cpython in https://bugs.chromium.org/p/chromium/issues/detail?id=1103275#c12?

@acoutts acoutts mentioned this issue Dec 5, 2020
Closed
4 tasks
@acoutts
Copy link

acoutts commented Apr 21, 2021

This appears to be fixed now.

@a-siva
Copy link
Contributor

a-siva commented Apr 21, 2021

Marking as fixed.

@a-siva a-siva closed this as completed Apr 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area-vm Use area-vm for VM related issues, including code coverage, FFI, and the AOT and JIT backends. os-osx
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@jmagman @alexmarkley @lrhn @mkustermann @Sunbreak @a-siva @acoutts