Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pyproject.toml-based pixi support #134

Merged
merged 1 commit into from
Apr 5, 2024

Conversation

traversaro
Copy link
Contributor

@traversaro traversaro commented Apr 3, 2024

Just an experiment, as pixi 0.18.0 was released today with some convenient features:

  • Just use pyproject.toml, no need for a separate pixi.toml file
  • Support for editable install

Furthermore, the PR make extensive use of the multiple environments feature (see https://prefix.dev/blog/introducing_multi_env_pixi) to permit to separate run and test dependencies, and separate environments for gpu support (just on linux-64). In a nutshell the following 4 environments are available:

  • default : the default environment, useful to run jaxsim on CPU
  • test-cpu : environment with test dependencies, to test jaxsim on CPU
  • gpu : environment with cuda dependencies, to run jaxsim on CPU or GPU
  • test-gpu : environment with test and cuda dependencies, to test jaxsim on GPU

To check that this is actually working as intended, run import jax;print(jax.devices()) in the different environments:

traversaro@IITBMP014LW012:~/jaxsim$ pixi run -e default python -c "import jax;print(jax.devices())"
An NVIDIA GPU may be present on this machine, but a CUDA-enabled jaxlib is not installed. Falling back to cpu.
[CpuDevice(id=0)]
traversaro@IITBMP014LW012:~/jaxsim$ pixi run -e gpu python -c "import jax;print(jax.devices())"
[cuda(id=0)]

To run the tests, run:

pixi run test

If you are on machine with CUDA support, a menu will ask if you want to run test in test-cpu or test-gpu environment, otherwise the test will run automatically in test-cpu as a fallback.

No CI/docs for now as I first need to understand if it is actually useful.

To test this, make sure that you are using the latest pixi 0.18.0 .

I am not super excited by the size of the pixi.lock, but I guess it is the price to pay for full reproducibility and the use of multiple environments on multiple operating systems. Probably we can reduce this by removing some operating systems from the platform list.


📚 Documentation preview 📚: https://jaxsim--134.org.readthedocs.build//134/

Copy link
Member

@diegoferigo diegoferigo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is super cool @traversaro! I have some curiosity, left some comment.

I'm not super happy about the size of the lock file either, but let's see how it evolves. For the time being, do we have any use-case for arm machines? If we want to slim it down, maybe we can leave them out at least for now.

Last curiosity: how should the lock file be updated? Do we have to do it manually once in a while (or better, automate it every one/two weeks)?

pyproject.toml Outdated
Comment on lines 76 to 88
[tool.pixi.dependencies]
coloredlogs = "*"
jax = "*"
jaxlib = "*"
jaxlie = "*"
jax-dataclasses = "*"
pptree = "*"
rod = "*"
typing_extensions = "*"
lxml = "*"
mediapy = "*"
mujoco = "*"
libgz-sim8 = "*"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we specify here the minimum versions as setup.cfg?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest, I wanted to avoid having that information duplicated. We can probably just add a pip check somewhere before the tests to check if we are violating the setup.cfg constraints, and only add the constraint if necessary.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That sounds good! I most cases, I guess that the solved environments are new enough.

pyproject.toml Outdated
Comment on lines 102 to 103
[tool.pixi.feature.gpu]

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
[tool.pixi.feature.gpu]
[tool.pixi.feature.gpu]

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that there is no change?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit, just blank line.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The suggestion is not removing the blank file, probably there is a bug in GitHub's suggestion method.

system-requirements = {cuda = "12.1"}
# cuda-nvcc and cuda-cupti are just a workaround for
# https://github.com/conda-forge/jaxlib-feedstock/pull/241
dependencies = {cuda-version = "12.*", cuda-nvcc = "*", cuda-cupti = "*", jaxlib = "* *cuda*"}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we pin a minimum jaxlib version in tool.pixi.dependency, should we also copy it here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess so.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I was wondering whether we have to duplicate it if a range is specified.

@traversaro
Copy link
Contributor Author

Last curiosity: how should the lock file be updated? Do we have to do it manually once in a while (or better, automate it every one/two weeks)?

I added a CI to do it automatically in robotology/whole-body-estimators#178, see also the documentation in prefix-dev/pixi#909 . However, given the size of the pixi.lock I am not sure if such a frequent update is a good idea now.

@diegoferigo
Copy link
Member

diegoferigo commented Apr 4, 2024

Last curiosity: how should the lock file be updated? Do we have to do it manually once in a while (or better, automate it every one/two weeks)?

I added a CI to do it automatically in robotology/whole-body-estimators#178, see also the documentation in prefix-dev/pixi#909 . However, given the size of the pixi.lock I am not sure if such a frequent update is a good idea now.

Thanks for the explanation! Given the youth of the tool, we can consider pixi support as experimental. Updating it manually when needed works for me.

@diegoferigo
Copy link
Member

Wow the lock is about 1MB, pretty large file considering that the entire git repository after a couple of years of development is just 2.8 MB 😅

@traversaro
Copy link
Contributor Author

Ok, I addressed the reviews. Leaving just linux-64 and depending on sdformat14 instead of gz-sim8 (as gz sdf is simply provided by sdformat14) I was able to reduce the pixi.lock size to 280KB, still not so small.

@traversaro
Copy link
Contributor Author

Ok, I addressed the reviews. Leaving just linux-64 and depending on sdformat14 instead of gz-sim8 (as gz sdf is simply provided by sdformat14) I was able to reduce the pixi.lock size to 280KB, still not so small.

Having just a default and gpu environment instead of default,test-cpu,gpu,test-gpu the size is reduced to 230 Kb.

@diegoferigo
Copy link
Member

Ok, I addressed the reviews. Leaving just linux-64 and depending on sdformat14 instead of gz-sim8 (as gz sdf is simply provided by sdformat14) I was able to reduce the pixi.lock size to 280KB, still not so small.

Perfect 😍

as gz sdf is simply provided by sdformat14

This is amazing! Since when? Last time I checked, gz was provided by Gazebo Sim and gz sdf was provided by a plugin from sdformat.

@traversaro
Copy link
Contributor Author

as gz sdf is simply provided by sdformat14

This is amazing! Since when? Last time I checked, gz was provided by Gazebo Sim and gz sdf was provided by a plugin from sdformat.

Since the beginning I guess? gz/ign as always been provided by gz-tools/ign-tools (see https://github.com/gazebosim/gz-tools), and then all the packages install the plugins with their tools, including gz-sim that installs the gz sim command.

@diegoferigo
Copy link
Member

as gz sdf is simply provided by sdformat14

This is amazing! Since when? Last time I checked, gz was provided by Gazebo Sim and gz sdf was provided by a plugin from sdformat.

Since the beginning I guess? gz/ign as always been provided by gz-tools/ign-tools (see https://github.com/gazebosim/gz-tools), and then all the packages install the plugins with their tools, including gz-sim that installs the gz sim command.

Ok, probably I overlooked things in the past. I always installed gz-sim* in my recent environments for nothing :) This also means that we can keep using gz sdf in ami-iit/rod#30.

@diegoferigo
Copy link
Member

Is this PR ready for merging?

@traversaro
Copy link
Contributor Author

Is this PR ready for merging?

Ready for me. Not super excited of the 200KB of space used, but I think the tradeoff with the added usability is worth it.

@tdejager
Copy link

tdejager commented Apr 5, 2024

Hopefully this will not explode the repo too much, because the file is diff-able so only changes should be recorded. However, it's still an initial add of 200KB of space for sure.

@diegoferigo
Copy link
Member

Yep I agree, 200KB of a diff-able file is not too bad, especially if compared to the initial one 1MB large. Let's start playing with this real-case example of JaxSim, which is pretty interesting due to the need of GPU environments, and then re-evaluate it in a few months.

I'm particularly interested in the definition of a custom command that in a single line creates the CPU/GPU environment and then opens one of the existing jupyter notebooks with the examples. It would make the life of external users interested in playing with JaxSim much easier.

@traversaro
Copy link
Contributor Author

I'm particularly interested in the definition of a custom command that in a single line creates the CPU/GPU environment and then opens one of the existing jupyter notebooks with the examples. It would make the life of external users interested in playing with JaxSim much easier.

We are almost there I guess, we need to add notebook as a dependency and add a task like https://github.com/traversaro/mujoco-mjx-pixi-example/blob/main/pixi.toml#L13 .

@diegoferigo diegoferigo merged commit 10c7199 into ami-iit:main Apr 5, 2024
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants