respect pyproject.toml

[Roger-luo]

I'm wondering if we can instead of creating a juliapkg.json, but putting a field inside pyproject.toml like

[julia.dependencies]
julia = "1.8"
Example = "0.5"
PythonCall = "0.8"

this would making packaging a lot easier because pyproject.toml will be saved in a package wheel.

[cjdoris]

Can you reliably find all the pyproject.toml files of all installed packages, regardless of how they were installed (from wheels or source)?

[MilesCranmer]

+1 for this, I also think as a longterm goal this would feel much more in-line with best practices to be able to specify this info there. Although @Roger-luo I think the syntax should be modified (see below)

@cjdoris the the role of pyproject.toml is just as a build configuration file, so it actually does not included in the installed package. So it would require a bit of additional config; see below. (While it is the "correct" option, I guess I'm also not sure the extra effort is worth it though...)

I think what would make sense is to have something like what setuptools_scm does:

[build-system]
requires = ["setuptools", "juliapkg_setup"]
build-backend = "setuptools.build_meta"

[project]
name = "myproject"
version = "0.1.0"
dependencies = [
  "juliapkg",
  "juliacall",
  # other python dependencies
]

[tool.juliapkg]
julia_requires = "~1.6.7, ~1.7, ~1.8, ~1.9, =1.10.0, ^1.10.3"
load_file = "src/julia_deps.py"  # Is generated during the build step

[tool.juliapkg.packages]
SymbolicRegression = { uuid = "8254be44-1295-4e6a-a16d-46603ac705cb", version = "=0.24.4" }
Serialization = { uuid = "9e88b42a-f829-5b0c-bbe9-9e923198166b", version = "1" }

which would generate src/julia_deps.py at build time which would contain the following:

import juliapkg

juliapkg.require_julia("~1.6.7, ~1.7, ~1.8, ~1.9, =1.10.0, ^1.10.3")
juliapkg.add("SymbolicRegression", "8254be44-1295-4e6a-a16d-46603ac705cb", version="=0.24.4")
juliapkg.add("Serialization", "9e88b42a-f829-5b0c-bbe9-9e923198166b", version="1")

Then, the package developer would add the following to the top of their src/__init__.py file:

import .julia_deps
...

This is the same way that version information is usually passed from pyproject.toml to the installed Python library using https://github.com/pypa/setuptools_scm. For example in my PySR package you can see I import this file version.py that doesn't actually exist yet:

# This file is created by setuptools_scm during the build process:
from .version import __version__

I think we could do something similar for pyjuliapkg, since we would want to transfer build information from the pyproject.toml to the Python runtime.

[cjdoris]

This is a neat idea. I've wondered about doing the analogue over in CondaPkg too - namely putting the dependencies into Project.toml instead of CondaPkg.toml. The situation is easier there because CondaPkg.toml is automatically gets bundled into the package it's in.

Instead of generating a julia_deps.py file (you're not supposed to call juliapkg.add every time you start julia, and anyway it would have no effect if juliapkg had already resolved first), I think it should just generate a juliapkg.json file. This way we can keep the existing functionality for finding dependencies.

Feel free to develop this (it would be a new package, which is convenient) - but it's not a thing I'll have time to work on.

[cjdoris]

One potential down-side - would it tie you into using a particular build system? There are quite a few Python build systems nowadays and I wouldn't want to force people into using only one. Taking setuptools_scm as the prototype - does that support build systems other than setuptools?

[MilesCranmer]

Good points.

Actually after reading more about it, while my suggestion was technically the "best practices" way of doing it, it just makes it so so so much more complicated for both us and the users.

I think the better option is to simply package pyproject.toml with Python and read it at import (just like juliapkg.json). For this, apparently all we would need is for users to specify a MANIFEST.in file in the root of their repo with:

include pyproject.toml

Then we could just have juliapkg read from that.

[MilesCranmer]

Thinking about this more, I do wonder if the Julia installation should happen during pip install rather than import. Then this sort of thing would already work with the original syntax @Roger-luo posted above (which I think is much nicer).

This is also how Cython packages get built — they compile during the pip install rather than at import time.

[Dale-Black]

I personally think adding Julia at pip install would be a beneficial change. Its the exact same amount of time in total, but waiting a while for a pip install feels a lot more normal than waiting for an import call to finish installing Julia.

[MilesCranmer]

@cjdoris the Rust integration tool for Python Maturin also does something like this: https://www.maturin.rs/

Mixed rust/python projects

To create a mixed rust/python project, create a folder with your module name (i.e. lib.name in Cargo.toml) next to your Cargo.toml and add your python sources there:

my-project
├── Cargo.toml
├── my_project
│   ├── __init__.py
│   └── bar.py
├── pyproject.toml
├── README.md
└── src
    └── lib.rs

You can specify a different python source directory in pyproject.toml by setting tool.maturin.python-source, for example

pyproject.toml

[tool.maturin]
python-source = "python"
module-name = "my_project._lib_name"

then the project structure would look like this:

my-project
├── Cargo.toml
├── python
│   └── my_project
│       ├── __init__.py
│       └── bar.py
├── pyproject.toml
├── README.md
└── src
    └── lib.rs

with the build backend specified in the pyproject.toml:

[build-system]
requires = ["maturin>=1.0,<2.0"]
build-backend = "maturin"

[cjdoris]

I don't think having Julia packages install at pip-install time is going to be possible, given it needs to happen knowing all the dependencies of all python packages in your environment. I think this would require custom behaviour in the python package manager. Doing the install at run time avoids this because we know all the packages are already installed.

[cjdoris]

As for putting the info into pyproject.toml, it's a nice idea, but we need a mechanism for that info to be put into the package when it is built - pyproject.toml itself is not bundled in a python package.

Ideally this would construct a juliapkg.toml file in the right place. I don't know how to achieve this. If it requires a custom build backend I think that's a non-starter. Maybe we can write plugins for existing build backends.

[MilesCranmer]

I don't think having Julia packages install at pip-install time is going to be possible, given it needs to happen knowing all the dependencies of all python packages in your environment.

Hm, is this actually an issue? I can add and remove packages from Julia environments, so wouldn't installing a new package be equivalent to add'ing a new Julia dependency? It could simply update the Project.toml in the julia_env.

Then if someone chooses to manually tweak the environment via juliapkg, that would also update the Project.toml. But pip installing a package would do this automatically, at install time.

---

I think something like maturin is a nice model for us: https://github.com/PyO3/maturin.

When using pip install on a package, pip tries to find a matching wheel and install that. If it doesn't find one, it downloads the source distribution and builds a wheel for the current platform, which requires the right compilers to be installed. Installing a wheel is much faster than installing a source distribution as building wheels is generally slow.

(Since juliaup is rust-based, we could even look at depending on maturin to pull this off...)

[cjdoris]

Hm, is this actually an issue? I can add and remove packages from Julia environments, so wouldn't installing a new package be equivalent to add'ing a new Julia dependency? It could simply update the Project.toml in the julia_env.

Sorry I don't follow. If I do pip install pysr (for example) what mechanism in juliapkg could automatically immediately install Julia, PythonCall and SymbolicRegression needed by PySR and JuliaCall? Can you be precise because I don't see how this can happen with existing standard tools like pip.

---

Maturin is a build backend isn't it? What exactly are you proposing to do with it?

(Apologies if I'm being slow.)

[MilesCranmer]

Hm, is this actually an issue? I can add and remove packages from Julia environments, so wouldn't installing a new package be equivalent to add'ing a new Julia dependency? It could simply update the Project.toml in the julia_env.

Sorry I don't follow. If I do pip install pysr (for example) what mechanism in juliapkg could automatically immediately install Julia, PythonCall and SymbolicRegression needed by PySR and JuliaCall? Can you be precise because I don't see how this can happen with existing standard tools like pip.

Sorry, what I meant was, we could have something like setuptools-rust: https://setuptools-rust.readthedocs.io/en/v1.1.2/ that gets installed before the setup is complete:

setuptools-rust is declared under build-system.requires within pyproject.toml:

[build-system]
requires = ["setuptools", "setuptools-rust"]
build-backend = "setuptools.build_meta"

and then is used within setup.py to declare the rust binary:

from setuptools import setup
from setuptools_rust import Binding, RustExtension

setup(
    name="hello-rust",
    version="1.0",
    rust_extensions=[RustExtension("hello_rust.hello_rust", binding=Binding.PyO3)],
    packages=["hello_rust"],
    zip_safe=False,
)

The Julia version of this, would, upon import in setup.py, do the equivalent of what pyjuliapkg does now, and read the pyproject.toml (which is accessible from setup.py).

So we could have:

[build-system]
requires = ["setuptools", "juliapkg"]
build-backend = "setuptools.build_meta"

and then something like the following within my setup.py:

from setuptools import setup
import juliapkg

setup(
    julia_env=[juliapkg.fetch_environment()]
)  # Would call juliapkg.resolve()

And this could read from pyproject.toml (first) and then juliapkg.json if not available, and install Julia. Similar to how Cython is compiled as part of the setup step.

Since pyproject.toml is accessible to setup.py, we could have all the dependency information there.

[cjdoris]

Ok thanks, I understand you now.

I think this approach to try and install Julia at pip-install time would only work with a source distribution of the package, not a wheel, because hooks like this can only run at build time. A wheel (which is the preferred way to distribute python packages) can't run arbitrary code at install time AFAIU.

Another issue is that I think this would end up doing juliapkg.resolve() for every package you install that uses this functionality. This seems quite unergonomic and possibly slow if it needs to keep changing the versions of Julia packages each time to satisfy different dependencies.

None of this totally blocks the idea, but preventing package authors from distributing wheels doesn't seem nice.

However we could put aside the idea of installing Julia dependencies at pip-install time. Instead I do think we can use this mechanism to read deps from the [tools.juliapkg] table in pyproject.toml and automatically write out a juliapkg.json file which is automatically distributed with the package.

I haven't looked at setuptools, but in hatchling (another build backend) it looks quite easy to write a plugin which does this: https://hatch.pypa.io/1.9/plugins/build-hook/reference/. Could start there and extend to other backends.