Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add mujoco package #2624

Open
josephrocca opened this issue May 27, 2022 · 10 comments
Open

Add mujoco package #2624

josephrocca opened this issue May 27, 2022 · 10 comments

Comments

@josephrocca
Copy link
Contributor

🐍 Package Request

DeepMind recently bought the MuJoCo company and open-sourced it. They also released official (pybind11) Python bindings. It's an extremely powerful engine, aimed specifically at scientific/robotics applications. Most other engines are focused on entertainment/games. Getting this to work in the browser would be beyond awesome.

I'm guessing the opengl stuff would be hard to add (unless someone has already worked on getting opengl-->webgl working), and if that's true, then ideally we can just skip it. Not sure how "entangled" it is with the codebase.

@josephrocca
Copy link
Contributor Author

josephrocca commented May 27, 2022

Here's my initial attempt. This is the first time I've tried this - I'm following these instructions, but it seems that they're missing a bunch of steps, so I've tried to improvise, which may mean I'm not doing things optimally or correctly:

cd /home/joe/Downloads
git clone https://github.com/pyodide/pyodide
cd pyodide
docker build -t pyodide .
docker run --rm -it -v /home/joe/Downloads:/stuff -w /stuff pyodide:latest bash
cd pyodide
make
python -m pyodide_build mkpkg mujoco
cd ./packages/mujoco
wget -qO- https://github.com/deepmind/mujoco/releases/download/2.2.0/mujoco-2.2.0-linux-x86_64.tar.gz | tar -xvz -C /stuff
# Add `env["MUJOCO_PATH"] = "/stuff/mujoco-2.2.0"` to `compile` function in /stuff/pyodide/pyodide-build/pyodide_build/pywasmcross.py
python -m pyodide_build buildpkg meta.yaml

The MUJOCO_PATH environment variable must point to the "binary release" according to the mujoco build instructions. That's what the wget command and the pywasmcross.py edit is for. But after tar -xvz it's a folder, rather than a binary file, so I'm not sure whether I've done that correctly.

And here's the not-very-helpful error that I get from cmake after running the final line in the above code block:

Click to expand
[2022-05-27 00:31:19] Building package mujoco...
* Creating virtualenv isolated environment...
* Installing packages in isolated environment... (setuptools >= 40.8.0, wheel)
* Installing packages in isolated environment... (cython, pythran)
* Getting dependencies for wheel...
running egg_info
writing mujoco.egg-info/PKG-INFO
writing dependency_links to mujoco.egg-info/dependency_links.txt
writing requirements to mujoco.egg-info/requires.txt
writing top-level names to mujoco.egg-info/top_level.txt
reading manifest file 'mujoco.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
adding license file 'LICENSES_THIRD_PARTY.md'
writing manifest file 'mujoco.egg-info/SOURCES.txt'
* Installing packages in isolated environment... (wheel)
* Installing packages in isolated environment... (cython, pythran)
* Building wheel...
running bdist_wheel
running build
running build_py
creating build
creating build/lib.emscripten_2_0_27_wasm32-cpython-310
creating build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco
copying mujoco/__init__.py -> build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco
copying mujoco/rollout_test.py -> build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco
copying mujoco/bindings_test.py -> build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco
copying mujoco/rollout.py -> build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco
copying mujoco/render_test.py -> build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco
creating build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco/osmesa
copying mujoco/osmesa/__init__.py -> build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco/osmesa
creating build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco/egl
copying mujoco/egl/__init__.py -> build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco/egl
copying mujoco/egl/egl_ext.py -> build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco/egl
creating build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco/glfw
copying mujoco/glfw/__init__.py -> build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco/glfw
running build_ext
CMake Error: The source directory "/stuff/pyodide/packages/mujoco/build/mujoco-2.2.0/build/temp.emscripten_2_0_27_wasm32-cpython-310/mujoco" does not exist.
Specify --help for usage, or press the help button on the CMake GUI.
Configuring CMake with the following arguments:
    -DPython3_ROOT_DIR:PATH=/tmp/build-env-3gr__99a
    -DPython3_EXECUTABLE:STRING=/tmp/build-env-3gr__99a/bin/python
    -DCMAKE_MODULE_PATH:PATH=cmake
    -DCMAKE_BUILD_TYPE:STRING=Release
    -DCMAKE_LIBRARY_OUTPUT_DIRECTORY:PATH=build/temp.emscripten_2_0_27_wasm32-cpython-310
    -DCMAKE_INTERPROCEDURAL_OPTIMIZATION=:BOOLON
    -DCMAKE_Fortran_COMPILER:STRING=
    -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON
    -DBUILD_TESTING:BOOL=OFF
    -DMUJOCO_LIBRARY_DIR:PATH=/stuff/mujoco-2.2.0/lib
    -DMUJOCO_INCLUDE_DIR:PATH=/stuff/mujoco-2.2.0/include
    -DPython3_LIBRARY=/usr/local/lib/python3.10
    -DPython3_INCLUDE_DIR=/usr/local/include/python3.10
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 363, in <module>
    main()
  File "/usr/local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 345, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/usr/local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 261, in build_wheel
    return _build_backend().build_wheel(wheel_directory, config_settings,
  File "/tmp/build-env-3gr__99a/lib/python3.10/site-packages/setuptools/build_meta.py", line 244, in build_wheel
    return self._build_with_temp_dir(['bdist_wheel'], '.whl',
  File "/tmp/build-env-3gr__99a/lib/python3.10/site-packages/setuptools/build_meta.py", line 229, in _build_with_temp_dir
    self.run_setup()
  File "/tmp/build-env-3gr__99a/lib/python3.10/site-packages/setuptools/build_meta.py", line 281, in run_setup
    super(_BuildMetaLegacyBackend,
  File "/tmp/build-env-3gr__99a/lib/python3.10/site-packages/setuptools/build_meta.py", line 174, in run_setup
    exec(compile(code, __file__, 'exec'), locals())
  File "setup.py", line 265, in <module>
    setup(
  File "/tmp/build-env-3gr__99a/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
  File "/tmp/build-env-3gr__99a/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 148, in setup
    return run_commands(dist)
  File "/tmp/build-env-3gr__99a/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
    dist.run_commands()
  File "/tmp/build-env-3gr__99a/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
    self.run_command(cmd)
  File "/tmp/build-env-3gr__99a/lib/python3.10/site-packages/setuptools/dist.py", line 1229, in run_command
    super().run_command(command)
  File "/tmp/build-env-3gr__99a/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/tmp/build-env-3gr__99a/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 299, in run
    self.run_command('build')
  File "/tmp/build-env-3gr__99a/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/tmp/build-env-3gr__99a/lib/python3.10/site-packages/setuptools/dist.py", line 1229, in run_command
    super().run_command(command)
  File "/tmp/build-env-3gr__99a/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/tmp/build-env-3gr__99a/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 136, in run
    self.run_command(cmd_name)
  File "/tmp/build-env-3gr__99a/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/tmp/build-env-3gr__99a/lib/python3.10/site-packages/setuptools/dist.py", line 1229, in run_command
    super().run_command(command)
  File "/tmp/build-env-3gr__99a/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "setup.py", line 144, in run
    self._configure_cmake()
  File "setup.py", line 237, in _configure_cmake
    subprocess.check_call(
  File "/usr/local/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '-DPython3_ROOT_DIR:PATH=/tmp/build-env-3gr__99a', '-DPython3_EXECUTABLE:STRING=/tmp/build-env-3gr__99a/bin/python', '-DCMAKE_MODULE_PATH:PATH=cmake', '-DCMAKE_BUILD_TYPE:STRING=Release', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY:PATH=build/temp.emscripten_2_0_27_wasm32-cpython-310', '-DCMAKE_INTERPROCEDURAL_OPTIMIZATION=:BOOLON', '-DCMAKE_Fortran_COMPILER:STRING=', '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON', '-DBUILD_TESTING:BOOL=OFF', '-DMUJOCO_LIBRARY_DIR:PATH=/stuff/mujoco-2.2.0/lib', '-DMUJOCO_INCLUDE_DIR:PATH=/stuff/mujoco-2.2.0/include', '-DPython3_LIBRARY=/usr/local/lib/python3.10', '-DPython3_INCLUDE_DIR=/usr/local/include/python3.10', 'mujoco']' returned non-zero exit status 1.

ERROR Backend subproccess exited when trying to invoke build_wheel
[2022-05-27 00:31:33] Succeeded building package mujoco in 13.6 seconds.

@hoodmane
Copy link
Member

hoodmane commented May 27, 2022

they're missing a bunch of steps

Would be nice to improve this if we ever get around to it. If you get ideas, PRs to improve the docs are welcome (though we ought to work on it ourselves).

docker run --rm -it -v /home/joe/Downloads:/stuff -w /stuff pyodide:latest bash

Did you try using our run_docker script?

cd ./packages/mujoco
wget -qO- https://github.com/deepmind/mujoco/releases/download/2.2.0/mujoco-2.2.0-linux-x86_64.tar.gz | tar -xvz -C /stuff
env["MUJOCO_PATH"] = "/stuff/mujoco-2.2.0"

This can go in the meta.yaml file:

build:
  script: |
    wget -qO- https://github.com/deepmind/mujoco/releases/download/2.2.0/mujoco-2.2.0-linux-x86_64.tar.gz | tar -xvz -C stuff
    export MUJOCO_PATH=$(pwd)/stuff

But most likely MUJOCO should be made into a separate library.

not-very-helpful cmake error

I find CMake completely baffling too. Maybe @ryanking13 or @henryiii can help.

@josephrocca
Copy link
Contributor Author

Wondering how to get cmake error to be more helpful. I've tried this:

build:
  script: |
    export CMAKE_ARGS="-Wdev"

Since -Wdev apparently enables developer warnings. But it still outputs the same unhelpful message:

subprocess.CalledProcessError: Command '['cmake', '-DPython3_ROOT_DIR:PATH=/tmp/build-env-xqz6d8a3', '-DPython3_EXECUTABLE:STRING=/tmp/build-env-xqz6d8a3/bin/python', '-DCMAKE_MODULE_PATH:PATH=cmake', '-DCMAKE_BUILD_TYPE:STRING=Release', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY:PATH=build/temp.emscripten_2_0_27_wasm32-cpython-310', '-DCMAKE_INTERPROCEDURAL_OPTIMIZATION=:BOOLON', '-DCMAKE_Fortran_COMPILER:STRING=', '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON', '-DBUILD_TESTING:BOOL=OFF', '-DMUJOCO_LIBRARY_DIR:PATH=/stuff/mujoco-2.2.0/lib', '-DMUJOCO_INCLUDE_DIR:PATH=/stuff/mujoco-2.2.0/include', '-DPython3_LIBRARY=/usr/local/lib/python3.10', '-DPython3_INCLUDE_DIR=/usr/local/include/python3.10', 'mujoco']' returned non-zero exit status 1.

I also tried adding a file /packages/<package-name>/cmake/Config.cmake like this with set(-Wdev) but that didn't work (which is unsurprising because I have no idea what I'm doing).

@henryiii
Copy link
Contributor

henryiii commented May 28, 2022

The error is this one:

CMake Error: The source directory "/stuff/pyodide/packages/mujoco/build/mujoco-2.2.0/build/temp.emscripten_2_0_27_wasm32-cpython-310/mujoco" does not exist.

It looks like the directory should have been made here: https://github.com/deepmind/mujoco/blob/e94a46c20fc7d64db6bd18edc1789007cab45641/python/setup.py#L228

You can set various trace options to get errors from CMake, but CMake isn't even starting, it's being given an non-existent source directory so it's immediately failing.

@henryiii
Copy link
Contributor

Hmm, this is the source directory:

https://github.com/deepmind/mujoco/blob/e94a46c20fc7d64db6bd18edc1789007cab45641/python/setup.py#L238

That doesn't exist. I notice this is a custom-rolled CMake adaptor, and is missing modern packaging tooling like pyproject.toml, and has specific instructions that include making an SDist first - https://github.com/deepmind/mujoco/tree/main/python. Are you following those instructions? It's quite possible it requires going through SDist first, and can't build directly from source.

@josephrocca
Copy link
Contributor Author

josephrocca commented May 29, 2022

Thanks for taking a look! My new meta.yaml after following the instructions in the mujoco readme looks like this:

package:
  name: mujoco
  version: 2.2.0
source:
  url: https://files.pythonhosted.org/packages/ad/52/3af0a5c1669eaf3bb81f3aeb04dfddceff9938c54f52033243428cc81e0d/mujoco-2.2.0.tar.gz
  sha256: 1bb370365a10f3e76eee8a438c8eafcd0d2721e05c1c9bf602cded8223f819cf
build:
  script: |
    wget --no-verbose https://github.com/deepmind/mujoco/releases/download/2.2.0/mujoco-2.2.0-linux-x86_64.tar.gz
    tar -xf mujoco-2.2.0-linux-x86_64.tar.gz
    export MUJOCO_PATH=$(pwd)/mujoco-2.2.0
    git clone https://github.com/deepmind/mujoco.git mujoco_repo
    cd mujoco_repo/python
    python3 -m venv /tmp/mujoco
    source /tmp/mujoco/bin/activate
    bash make_sdist.sh
    cd dist
    pip install mujoco-2.2.0.tar.gz
test:
  imports:
    - mujoco
about:
  home: https://github.com/deepmind/mujoco
  PyPI: https://pypi.org/project/mujoco
  summary: MuJoCo Physics Simulator
  license: Apache License 2.0

That still gives the same cmake error for some reason:

CMake Error: The source directory "/stuff/pyodide/packages/mujoco/build/mujoco-2.2.0/build/temp.emscripten_2_0_27_wasm32-cpython-310/mujoco" does not exist.

Full logs:

Click to expand
# python -m pyodide_build buildpkg meta.yaml

[2022-05-29 02:01:58] Building package mujoco...
2022-05-29 02:01:58 URL:https://objects.githubusercontent.com/github-production-release-asset-2e65be/400501136/0311c9c2-1787-4dc3-a492-a0dd373da867?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220529%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220529T020056Z&X-Amz-Expires=300&X-Amz-Signature=15f3b3d838f1793c6940e54946f91f0ad73cf77ae268c9d35c79fa2e08f2818a&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=400501136&response-content-disposition=attachment%3B%20filename%3Dmujoco-2.2.0-linux-x86_64.tar.gz&response-content-type=application%2Foctet-stream [4632601/4632601] -> "mujoco-2.2.0-linux-x86_64.tar.gz" [1]
Cloning into 'mujoco_repo'...
Requirement already satisfied: pip in /tmp/mujoco/lib/python3.10/site-packages (22.1.1)
Requirement already satisfied: setuptools in /tmp/mujoco/lib/python3.10/site-packages (62.3.2)
Requirement already satisfied: absl-py in /tmp/mujoco/lib/python3.10/site-packages (1.0.0)
Requirement already satisfied: six in /tmp/mujoco/lib/python3.10/site-packages (from absl-py) (1.16.0)
/tmp/tmp.cUqQu18UEW /stuff/pyodide/packages/mujoco/build/mujoco-2.2.0/mujoco_repo/python
running sdist
running egg_info
creating mujoco.egg-info
writing mujoco.egg-info/PKG-INFO
writing dependency_links to mujoco.egg-info/dependency_links.txt
writing requirements to mujoco.egg-info/requires.txt
writing top-level names to mujoco.egg-info/top_level.txt
writing manifest file 'mujoco.egg-info/SOURCES.txt'
reading manifest file 'mujoco.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'mujoco.egg-info/SOURCES.txt'
running check
creating mujoco-2.2.0
creating mujoco-2.2.0/cmake
creating mujoco-2.2.0/mujoco
creating mujoco-2.2.0/mujoco.egg-info
creating mujoco-2.2.0/mujoco/egl
creating mujoco-2.2.0/mujoco/glfw
creating mujoco-2.2.0/mujoco/osmesa
creating mujoco-2.2.0/mujoco/util
copying files to mujoco-2.2.0...
copying LICENSE -> mujoco-2.2.0
copying MANIFEST.in -> mujoco-2.2.0
copying README.md -> mujoco-2.2.0
copying setup.py -> mujoco-2.2.0
copying cmake/CheckAvxSupport.cmake -> mujoco-2.2.0/cmake
copying cmake/FindOrFetch.cmake -> mujoco-2.2.0/cmake
copying cmake/MujocoDependencies.cmake -> mujoco-2.2.0/cmake
copying cmake/MujocoHarden.cmake -> mujoco-2.2.0/cmake
copying cmake/MujocoLinkOptions.cmake -> mujoco-2.2.0/cmake
copying cmake/MujocoMacOS.cmake -> mujoco-2.2.0/cmake
copying cmake/MujocoOptions.cmake -> mujoco-2.2.0/cmake
copying cmake/ShellTests.cmake -> mujoco-2.2.0/cmake
copying cmake/TargetAddRpath.cmake -> mujoco-2.2.0/cmake
copying mujoco/CMakeLists.txt -> mujoco-2.2.0/mujoco
copying mujoco/__init__.py -> mujoco-2.2.0/mujoco
copying mujoco/bindings_test.py -> mujoco-2.2.0/mujoco
copying mujoco/callbacks.cc -> mujoco-2.2.0/mujoco
copying mujoco/constants.cc -> mujoco-2.2.0/mujoco
copying mujoco/enum_traits.h -> mujoco-2.2.0/mujoco
copying mujoco/enums.cc -> mujoco-2.2.0/mujoco
copying mujoco/errors.cc -> mujoco-2.2.0/mujoco
copying mujoco/errors.h -> mujoco-2.2.0/mujoco
copying mujoco/function_traits.h -> mujoco-2.2.0/mujoco
copying mujoco/functions.cc -> mujoco-2.2.0/mujoco
copying mujoco/functions.h -> mujoco-2.2.0/mujoco
copying mujoco/indexer_xmacro.h -> mujoco-2.2.0/mujoco
copying mujoco/indexers.cc -> mujoco-2.2.0/mujoco
copying mujoco/indexers.h -> mujoco-2.2.0/mujoco
copying mujoco/mjdata_meta.h -> mujoco-2.2.0/mujoco
copying mujoco/raw.h -> mujoco-2.2.0/mujoco
copying mujoco/render.cc -> mujoco-2.2.0/mujoco
copying mujoco/render_test.py -> mujoco-2.2.0/mujoco
copying mujoco/rollout.cc -> mujoco-2.2.0/mujoco
copying mujoco/rollout.py -> mujoco-2.2.0/mujoco
copying mujoco/rollout_test.py -> mujoco-2.2.0/mujoco
copying mujoco/serialization.h -> mujoco-2.2.0/mujoco
copying mujoco/structs.cc -> mujoco-2.2.0/mujoco
copying mujoco/structs.h -> mujoco-2.2.0/mujoco
copying mujoco.egg-info/PKG-INFO -> mujoco-2.2.0/mujoco.egg-info
copying mujoco.egg-info/SOURCES.txt -> mujoco-2.2.0/mujoco.egg-info
copying mujoco.egg-info/dependency_links.txt -> mujoco-2.2.0/mujoco.egg-info
copying mujoco.egg-info/requires.txt -> mujoco-2.2.0/mujoco.egg-info
copying mujoco.egg-info/top_level.txt -> mujoco-2.2.0/mujoco.egg-info
copying mujoco/egl/__init__.py -> mujoco-2.2.0/mujoco/egl
copying mujoco/egl/egl_ext.py -> mujoco-2.2.0/mujoco/egl
copying mujoco/glfw/__init__.py -> mujoco-2.2.0/mujoco/glfw
copying mujoco/osmesa/__init__.py -> mujoco-2.2.0/mujoco/osmesa
copying mujoco/util/CMakeLists.txt -> mujoco-2.2.0/mujoco/util
copying mujoco/util/array_traits.h -> mujoco-2.2.0/mujoco/util
copying mujoco/util/array_traits_test.cc -> mujoco-2.2.0/mujoco/util
copying mujoco/util/crossplatform.h -> mujoco-2.2.0/mujoco/util
copying mujoco/util/func_traits.h -> mujoco-2.2.0/mujoco/util
copying mujoco/util/func_traits_test.cc -> mujoco-2.2.0/mujoco/util
copying mujoco/util/func_wrap.h -> mujoco-2.2.0/mujoco/util
copying mujoco/util/func_wrap_test.cc -> mujoco-2.2.0/mujoco/util
copying mujoco/util/tuple_tools.h -> mujoco-2.2.0/mujoco/util
copying mujoco/util/tuple_tools_test.cc -> mujoco-2.2.0/mujoco/util
Writing mujoco-2.2.0/setup.cfg
creating dist
Creating tar archive
removing 'mujoco-2.2.0' (and everything under it)
mujoco-2.2.0/
mujoco-2.2.0/LICENSE
mujoco-2.2.0/MANIFEST.in
mujoco-2.2.0/PKG-INFO
mujoco-2.2.0/README.md
mujoco-2.2.0/cmake/
mujoco-2.2.0/cmake/CheckAvxSupport.cmake
mujoco-2.2.0/cmake/FindOrFetch.cmake
mujoco-2.2.0/cmake/MujocoDependencies.cmake
mujoco-2.2.0/cmake/MujocoHarden.cmake
mujoco-2.2.0/cmake/MujocoLinkOptions.cmake
mujoco-2.2.0/cmake/MujocoMacOS.cmake
mujoco-2.2.0/cmake/MujocoOptions.cmake
mujoco-2.2.0/cmake/ShellTests.cmake
mujoco-2.2.0/cmake/TargetAddRpath.cmake
mujoco-2.2.0/mujoco/
mujoco-2.2.0/mujoco/CMakeLists.txt
mujoco-2.2.0/mujoco/__init__.py
mujoco-2.2.0/mujoco/bindings_test.py
mujoco-2.2.0/mujoco/callbacks.cc
mujoco-2.2.0/mujoco/constants.cc
mujoco-2.2.0/mujoco/egl/
mujoco-2.2.0/mujoco/egl/__init__.py
mujoco-2.2.0/mujoco/egl/egl_ext.py
mujoco-2.2.0/mujoco/enum_traits.h
mujoco-2.2.0/mujoco/enums.cc
mujoco-2.2.0/mujoco/errors.cc
mujoco-2.2.0/mujoco/errors.h
mujoco-2.2.0/mujoco/function_traits.h
mujoco-2.2.0/mujoco/functions.cc
mujoco-2.2.0/mujoco/functions.h
mujoco-2.2.0/mujoco/glfw/
mujoco-2.2.0/mujoco/glfw/__init__.py
mujoco-2.2.0/mujoco/indexer_xmacro.h
mujoco-2.2.0/mujoco/indexers.cc
mujoco-2.2.0/mujoco/indexers.h
mujoco-2.2.0/mujoco/mjdata_meta.h
mujoco-2.2.0/mujoco/osmesa/
mujoco-2.2.0/mujoco/osmesa/__init__.py
mujoco-2.2.0/mujoco/raw.h
mujoco-2.2.0/mujoco/render.cc
mujoco-2.2.0/mujoco/render_test.py
mujoco-2.2.0/mujoco/rollout.cc
mujoco-2.2.0/mujoco/rollout.py
mujoco-2.2.0/mujoco/rollout_test.py
mujoco-2.2.0/mujoco/serialization.h
mujoco-2.2.0/mujoco/structs.cc
mujoco-2.2.0/mujoco/structs.h
mujoco-2.2.0/mujoco/util/
mujoco-2.2.0/mujoco/util/CMakeLists.txt
mujoco-2.2.0/mujoco/util/array_traits.h
mujoco-2.2.0/mujoco/util/array_traits_test.cc
mujoco-2.2.0/mujoco/util/crossplatform.h
mujoco-2.2.0/mujoco/util/func_traits.h
mujoco-2.2.0/mujoco/util/func_traits_test.cc
mujoco-2.2.0/mujoco/util/func_wrap.h
mujoco-2.2.0/mujoco/util/func_wrap_test.cc
mujoco-2.2.0/mujoco/util/tuple_tools.h
mujoco-2.2.0/mujoco/util/tuple_tools_test.cc
mujoco-2.2.0/mujoco.egg-info/
mujoco-2.2.0/mujoco.egg-info/PKG-INFO
mujoco-2.2.0/mujoco.egg-info/SOURCES.txt
mujoco-2.2.0/mujoco.egg-info/dependency_links.txt
mujoco-2.2.0/mujoco.egg-info/requires.txt
mujoco-2.2.0/mujoco.egg-info/top_level.txt
mujoco-2.2.0/setup.cfg
mujoco-2.2.0/setup.py
/stuff/pyodide/packages/mujoco/build/mujoco-2.2.0/mujoco_repo/python
Processing ./mujoco-2.2.0.tar.gz
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'done'
Requirement already satisfied: absl-py in /tmp/mujoco/lib/python3.10/site-packages (from mujoco==2.2.0) (1.0.0)
Requirement already satisfied: glfw in /tmp/mujoco/lib/python3.10/site-packages (from mujoco==2.2.0) (2.5.3)
Requirement already satisfied: numpy in /tmp/mujoco/lib/python3.10/site-packages (from mujoco==2.2.0) (1.22.4)
Requirement already satisfied: pyopengl in /tmp/mujoco/lib/python3.10/site-packages (from mujoco==2.2.0) (3.1.6)
Requirement already satisfied: six in /tmp/mujoco/lib/python3.10/site-packages (from absl-py->mujoco==2.2.0) (1.16.0)
Using legacy 'setup.py install' for mujoco, since package 'wheel' is not installed.
Installing collected packages: mujoco
  Attempting uninstall: mujoco
    Found existing installation: mujoco 2.2.0
    Uninstalling mujoco-2.2.0:
      Successfully uninstalled mujoco-2.2.0
  Running setup.py install for mujoco: started
  Running setup.py install for mujoco: still running...
  Running setup.py install for mujoco: finished with status 'done'
Successfully installed mujoco-2.2.0
* Creating virtualenv isolated environment...
* Installing packages in isolated environment... (setuptools >= 40.8.0, wheel)
* Installing packages in isolated environment... (cython, pythran)
* Getting dependencies for wheel...
running egg_info
writing mujoco.egg-info/PKG-INFO
writing dependency_links to mujoco.egg-info/dependency_links.txt
writing requirements to mujoco.egg-info/requires.txt
writing top-level names to mujoco.egg-info/top_level.txt
reading manifest file 'mujoco.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
adding license file 'LICENSES_THIRD_PARTY.md'
writing manifest file 'mujoco.egg-info/SOURCES.txt'
* Installing packages in isolated environment... (wheel)
* Installing packages in isolated environment... (cython, pythran)
* Building wheel...
running bdist_wheel
running build
running build_py
creating build
creating build/lib.emscripten_2_0_27_wasm32-cpython-310
creating build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco
copying mujoco/__init__.py -> build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco
copying mujoco/rollout_test.py -> build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco
copying mujoco/bindings_test.py -> build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco
copying mujoco/rollout.py -> build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco
copying mujoco/render_test.py -> build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco
creating build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco/osmesa
copying mujoco/osmesa/__init__.py -> build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco/osmesa
creating build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco/egl
copying mujoco/egl/__init__.py -> build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco/egl
copying mujoco/egl/egl_ext.py -> build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco/egl
creating build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco/glfw
copying mujoco/glfw/__init__.py -> build/lib.emscripten_2_0_27_wasm32-cpython-310/mujoco/glfw
running build_ext
CMake Error: The source directory "/stuff/pyodide/packages/mujoco/build/mujoco-2.2.0/build/temp.emscripten_2_0_27_wasm32-cpython-310/mujoco" does not exist.
Specify --help for usage, or press the help button on the CMake GUI.
Configuring CMake with the following arguments:
    -DPython3_ROOT_DIR:PATH=/tmp/build-env-564kotd1
    -DPython3_EXECUTABLE:STRING=/tmp/build-env-564kotd1/bin/python
    -DCMAKE_MODULE_PATH:PATH=cmake
    -DCMAKE_BUILD_TYPE:STRING=Release
    -DCMAKE_LIBRARY_OUTPUT_DIRECTORY:PATH=build/temp.emscripten_2_0_27_wasm32-cpython-310
    -DCMAKE_INTERPROCEDURAL_OPTIMIZATION=:BOOLON
    -DCMAKE_Fortran_COMPILER:STRING=
    -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON
    -DBUILD_TESTING:BOOL=OFF
    -DMUJOCO_LIBRARY_DIR:PATH=/stuff/pyodide/packages/mujoco/build/mujoco-2.2.0/mujoco-2.2.0/lib
    -DMUJOCO_INCLUDE_DIR:PATH=/stuff/pyodide/packages/mujoco/build/mujoco-2.2.0/mujoco-2.2.0/include
    -DPython3_LIBRARY=/usr/local/lib/python3.10
    -DPython3_INCLUDE_DIR=/usr/local/include/python3.10
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 363, in <module>
    main()
  File "/usr/local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 345, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
  File "/usr/local/lib/python3.10/site-packages/pep517/in_process/_in_process.py", line 261, in build_wheel
    return _build_backend().build_wheel(wheel_directory, config_settings,
  File "/tmp/build-env-564kotd1/lib/python3.10/site-packages/setuptools/build_meta.py", line 244, in build_wheel
    return self._build_with_temp_dir(['bdist_wheel'], '.whl',
  File "/tmp/build-env-564kotd1/lib/python3.10/site-packages/setuptools/build_meta.py", line 229, in _build_with_temp_dir
    self.run_setup()
  File "/tmp/build-env-564kotd1/lib/python3.10/site-packages/setuptools/build_meta.py", line 281, in run_setup
    super(_BuildMetaLegacyBackend,
  File "/tmp/build-env-564kotd1/lib/python3.10/site-packages/setuptools/build_meta.py", line 174, in run_setup
    exec(compile(code, __file__, 'exec'), locals())
  File "setup.py", line 265, in <module>
    setup(
  File "/tmp/build-env-564kotd1/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
    return distutils.core.setup(**attrs)
  File "/tmp/build-env-564kotd1/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 148, in setup
    return run_commands(dist)
  File "/tmp/build-env-564kotd1/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
    dist.run_commands()
  File "/tmp/build-env-564kotd1/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
    self.run_command(cmd)
  File "/tmp/build-env-564kotd1/lib/python3.10/site-packages/setuptools/dist.py", line 1229, in run_command
    super().run_command(command)
  File "/tmp/build-env-564kotd1/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/tmp/build-env-564kotd1/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 299, in run
    self.run_command('build')
  File "/tmp/build-env-564kotd1/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/tmp/build-env-564kotd1/lib/python3.10/site-packages/setuptools/dist.py", line 1229, in run_command
    super().run_command(command)
  File "/tmp/build-env-564kotd1/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "/tmp/build-env-564kotd1/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 136, in run
    self.run_command(cmd_name)
  File "/tmp/build-env-564kotd1/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/tmp/build-env-564kotd1/lib/python3.10/site-packages/setuptools/dist.py", line 1229, in run_command
    super().run_command(command)
  File "/tmp/build-env-564kotd1/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
    cmd_obj.run()
  File "setup.py", line 144, in run
    self._configure_cmake()
  File "setup.py", line 237, in _configure_cmake
    subprocess.check_call(
  File "/usr/local/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '-DPython3_ROOT_DIR:PATH=/tmp/build-env-564kotd1', '-DPython3_EXECUTABLE:STRING=/tmp/build-env-564kotd1/bin/python', '-DCMAKE_MODULE_PATH:PATH=cmake', '-DCMAKE_BUILD_TYPE:STRING=Release', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY:PATH=build/temp.emscripten_2_0_27_wasm32-cpython-310', '-DCMAKE_INTERPROCEDURAL_OPTIMIZATION=:BOOLON', '-DCMAKE_Fortran_COMPILER:STRING=', '-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON', '-DBUILD_TESTING:BOOL=OFF', '-DMUJOCO_LIBRARY_DIR:PATH=/stuff/pyodide/packages/mujoco/build/mujoco-2.2.0/mujoco-2.2.0/lib', '-DMUJOCO_INCLUDE_DIR:PATH=/stuff/pyodide/packages/mujoco/build/mujoco-2.2.0/mujoco-2.2.0/include', '-DPython3_LIBRARY=/usr/local/lib/python3.10', '-DPython3_INCLUDE_DIR=/usr/local/include/python3.10', 'mujoco']' returned non-zero exit status 1.

ERROR Backend subproccess exited when trying to invoke build_wheel
[2022-05-29 02:04:25] Succeeded building package mujoco in 147.8 seconds.

I'm wondering if the build.yml here has any clues, since presumably it would require very similar commands to what the Pyodide meta.yaml would need.

I think the meta problem here is that I don't understand how the whole Pyodide build process works. I've added some more info to the docs here based on stuff I learned while trying to get this working, but I think it could do with a high-level overview of the build process, and some added reasoning/understanding at each step instead of just the concrete commands/instructions. That would help newbies like me a lot in understanding what might be going wrong in cases like this - even simple stuff like what the source.url yaml property is used for (I think this is what confused me RE following the mujoco readme instructions - why git clone etc if source.url is already pointing to source code?), and what the packages/<package-name>/build directory should contain after building, what is the "final" thing that we're trying to produce with the build.script yaml property (i.e. what specifically are we trying to 'prep' for cmake), and so on.

It could just be that a task like this is a bit beyond my "zone of proximal development".

@hoodmane
Copy link
Member

hoodmane commented May 29, 2022

You need to build libmujoco.so first before you can build the Python bindings. I tried the following:

packages/libmujoco/meta.yaml
package:
  name: libmujoco
  version: 2.2.0
source:
  url: https://github.com/deepmind/mujoco/archive/refs/tags/2.2.0.tar.gz
  sha256: 1bb370365a10f3e76eee8a438c8eafcd0d2721e05c1c9bf602cded8223f819cf
build:
  sharedlibrary: true
  script: |
    mkdir build
    cd build
    emcmake cmake ..
    emmake make
test:
  imports:
    - mujoco
about:
  home: https://github.com/deepmind/mujoco
  PyPI: https://pypi.org/project/mujoco
  summary: MuJoCo Physics Simulator
  license: Apache License 2.0

This fails with: "Could NOT find required library LibM". I'm not sure why it's not finding the math library. libm is definitely available, for instance numpy finds it and successfully links it. I think we have to leave this question to someone else who understands CMake.

Anyways, for the Python package I have the following. Not sure if it will work.

packages/mujoco/meta.yaml
package:
  name: mujoco
  version: 2.2.0
source:
  url: https://github.com/deepmind/mujoco/archive/refs/tags/2.2.0.tar.gz
  # sha256: 1bb370365a10f3e76eee8a438c8eafcd0d2721e05c1c9bf602cded8223f819cf
  extras:
    - [setup.py, setup.py]
build:
  script: |
    export MUJOCO_PATH=$(pwd)/mujoco-2.2.0
    cd python
    python3 -m venv .venv
    source .venv/bin/activate
    bash make_sdist.sh
test:
  imports:
    - mujoco
about:
  home: https://github.com/deepmind/mujoco
  PyPI: https://pypi.org/project/mujoco
  summary: MuJoCo Physics Simulator
  license: Apache License 2.0
packages/mujoco/setup.py
from importlib.machinery import SourceFileLoader

# Just import the real setup.py in the python folder
SourceFileLoader("setup", "python/setup.py").load_module() 

@lshdlut
Copy link

lshdlut commented Aug 16, 2024

Sorry to bother but any progress on this?

@ryanking13
Copy link
Member

@lshdlut

No. But we can help debug the issue if you try to build the package yourselves and report what goes wrong.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants