ExecuTorch main branch setup fails while stable branch works

[kenzhang1118]

Follow the instruction at
https://pytorch.org/executorch/main/getting-started-setup.html

It failed at
(.executorch) (base) xiaokz@bcd07466b662 executorch % pip install . --no-build-isolation
Processing /Users/xiaokz/executorch_0308/executorch
Preparing metadata (pyproject.toml) ... error
error: subprocess-exited-with-error

× Preparing metadata (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [14 lines of output]
/Users/xiaokz/executorch_0308/executorch/.executorch/lib/python3.11/site-packages/setuptools/config/pyprojecttoml.py:108: _BetaConfiguration: Support for [tool.setuptools] in pyproject.toml is still beta.
warnings.warn(msg, _BetaConfiguration)
running dist_info
copying from schema/scalar_type.fbs to exir/_serialize/scalar_type.fbs
copying from schema/program.fbs to exir/_serialize/program.fbs
copying from sdk/bundled_program/schema/bundled_program_schema.fbs to sdk/bundled_program/serialize/bundled_program_schema.fbs
copying from sdk/bundled_program/schema/scalar_type.fbs to sdk/bundled_program/serialize/scalar_type.fbs
creating /private/var/folders/r9/b1wm6k_553d82xxqt5gm6xmm0000gr/T/pip-modern-metadata-jfim5m69/executorch.egg-info
writing /private/var/folders/r9/b1wm6k_553d82xxqt5gm6xmm0000gr/T/pip-modern-metadata-jfim5m69/executorch.egg-info/PKG-INFO
writing dependency_links to /private/var/folders/r9/b1wm6k_553d82xxqt5gm6xmm0000gr/T/pip-modern-metadata-jfim5m69/executorch.egg-info/dependency_links.txt
writing requirements to /private/var/folders/r9/b1wm6k_553d82xxqt5gm6xmm0000gr/T/pip-modern-metadata-jfim5m69/executorch.egg-info/requires.txt
writing top-level names to /private/var/folders/r9/b1wm6k_553d82xxqt5gm6xmm0000gr/T/pip-modern-metadata-jfim5m69/executorch.egg-info/top_level.txt
writing manifest file '/private/var/folders/r9/b1wm6k_553d82xxqt5gm6xmm0000gr/T/pip-modern-metadata-jfim5m69/executorch.egg-info/SOURCES.txt'
error: package directory 'backends/arm/third-party/serialization_lib/python/serializer' does not exist
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

However stable branch works fine
https://pytorch.org/executorch/stable/getting-started-setup.html

[mergennachin]

Did you try these steps?

git submodule sync
git submodule update --init

[kenzhang1118]

Yes, I have done these steps, and tried on two computers.

[mergennachin]

I just did a clone of the main branch and installation on a clean conda environment and it is working just fine for me.

[kenzhang1118]

using conda,

conda create -yn executorch python=3.10.0
conda activate executorch

instead of

python3 -m venv .executorch
source .executorch/bin/activate

./install_requirements.sh can finish

there are some errors like below:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
pandas 2.2.1 requires numpy<2,>=1.22.4; python_version < "3.11", but you have numpy 2.0.0b1 which is incompatible.

python3 -m examples.portable.scripts.export --model_name="add" works fine

Building runtime failed

/tmp/buck2 build //examples/portable/executor_runner:executor_runner --show-output
error as below:
(executorch) xiaokz@u84faae0ef1b250:~/execu/executorch$ /tmp/buck2 build //examples/portable/executor_runner:executor_runner --show-output
From load at implicit location

Caused by:
0: From load at third-party/prelude/prelude.bzl:8:6-29
1: From load at third-party/prelude/native.bzl:15:6-45
2: From load at third-party/prelude/apple/apple_macro_layer.bzl:8:6-32
3: Error evaluating module: prelude//apple/apple_bundle_config.bzl
4: error: Variable typing not found
--> third-party/prelude/apple/apple_bundle_config.bzl:14:40
|
14 | def apple_bundle_config() -> dict[str, typing.Any]:
| ^^^^^^
|

Build ID: d78a0afb-1e63-48e5-9d36-f372fa112850
Jobs completed: 3. Time elapsed: 0.0s.
BUILD FAILED

Try Cmake, also has errors:

(executorch) xiaokz@u84faae0ef1b250:~/execu/executorch/cmake-out$ cmake -DBUCK2=/tmp/buck2 ..
-- The C compiler identification is GNU 11.4.0
-- The CXX compiler identification is GNU 11.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- executorch: Generating source lists
-- executorch: Generating source file list /home/ANT.AMAZON.COM/xiaokz/execu/executorch/cmake-out/executorch_srcs.cmake
Error while generating /home/ANT.AMAZON.COM/xiaokz/execu/executorch/cmake-out/executorch_srcs.cmake. Exit code: 1
Output:

Error:
Traceback (most recent call last):
File "/home/ANT.AMAZON.COM/xiaokz/execu/executorch/build/extract_sources.py", line 79, in run
cp: subprocess.CompletedProcess = subprocess.run(
File "/home/ANT.AMAZON.COM/xiaokz/anaconda3/envs/executorch/lib/python3.10/subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/tmp/buck2', 'cquery', "inputs(deps('//runtime/executor:program'))"]' returned non-zero exit status 2.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/ANT.AMAZON.COM/xiaokz/execu/executorch/build/extract_sources.py", line 235, in
main()
File "/home/ANT.AMAZON.COM/xiaokz/execu/executorch/build/extract_sources.py", line 220, in main
target_to_srcs[name] = sorted(target.get_sources(graph, runner))
File "/home/ANT.AMAZON.COM/xiaokz/execu/executorch/build/extract_sources.py", line 133, in get_sources
sources: set[str] = set(runner.run(["cquery", query]))
File "/home/ANT.AMAZON.COM/xiaokz/execu/executorch/build/extract_sources.py", line 84, in run
raise RuntimeError(ex.stderr.decode("utf-8")) from ex
RuntimeError: [2024-03-13T20:43:05.877-07:00] Build ID: 78e55616-e21d-4cf8-bb7c-fcb7ae5aa61d
[2024-03-13T20:43:05.877-07:00] File changed: root//cmake-out
[2024-03-13T20:43:05.877-07:00] File changed: root//cmake-out/CMakeFiles
[2024-03-13T20:43:05.877-07:00] File changed: root//cmake-out/CMakeFiles/CMakeOutput.log
[2024-03-13T20:43:05.877-07:00] 82 additional file change events
[2024-03-13T20:43:05.889-07:00] Command failed:
[2024-03-13T20:43:05.889-07:00] From load at implicit location
[2024-03-13T20:43:05.889-07:00]
[2024-03-13T20:43:05.889-07:00] Caused by:
[2024-03-13T20:43:05.889-07:00] 0: From load at third-party/prelude/prelude.bzl:8:6-29
[2024-03-13T20:43:05.889-07:00] 1: From load at third-party/prelude/native.bzl:15:6-45
[2024-03-13T20:43:05.889-07:00] 2: From load at third-party/prelude/apple/apple_macro_layer.bzl:8:6-32
[2024-03-13T20:43:05.889-07:00] 3: Error evaluating module: prelude//apple/apple_bundle_config.bzl
[2024-03-13T20:43:05.889-07:00] 4: error: Variable typing not found
[2024-03-13T20:43:05.889-07:00] --> third-party/prelude/apple/apple_bundle_config.bzl:14:40
[2024-03-13T20:43:05.889-07:00] |
[2024-03-13T20:43:05.889-07:00] 14 | def apple_bundle_config() -> dict[str, typing.Any]:
[2024-03-13T20:43:05.889-07:00] | ^^^^^^
[2024-03-13T20:43:05.889-07:00] |
[2024-03-13T20:43:05.889-07:00]

CMake Error at build/Utils.cmake:116 (message):
executorch: source list generation failed
Call Stack (most recent call first):
CMakeLists.txt:181 (extract_sources)

[mergennachin]

Hi @kenzhang1118

We just stabilized and revamped our getting started page. https://pytorch.org/executorch/stable/getting-started-setup.html

Please re-open the issue, if you still encounter this error.