Add pyjuliacall pyjuliapkg via grayskull

[mkitti]

Checklist

• Title of this PR is meaningful: e.g. "Adding my_nifty_package", not "updated meta.yaml".
• License file is packaged (see here for an example).
• Source is from official source.
• Package does not vendor other packages. (If a package uses the source of another package, they should be separate packages or the licenses of all packages need to be packaged).
• If static libraries are linked in, the license of the static library is packaged.
• Package does not ship static libraries. If static libraries are needed, follow CFEP-18.
• Build number is 0.
• A tarball (url) rather than a repo (e.g. git_url) is used in your recipe (see here for more details).
• GitHub users listed in the maintainer section have posted a comment confirming they are willing to be listed there.
• When in trouble, please check our knowledge base documentation before pinging a team.

@conda-forge/help-python @conda-forge/help-julia

This pull request packages julicall and [juliapkg] from pypi as pyjuliacall and pyjuliapkg respectively. pyjuliapkg is a dependency of pyjuliacall.

There also exists a R package called JuliaCall on CRAN.

The "py" prefix is added to disambiguate these packages from the R package with a similar name and potential Julia packages following the deconfliction protocol of
conda-forge/conda-forge.github.io#18 (comment) . Also note that the upstream source repository of juliapkg is called pyjuliapkg.

This pull request is meant so supersede #20379 and #20380 which are both now stale. The main difference between this pull request and those ones is that the added "py" prefix.

The upstream source repositories are as follows.

• https://github.com/JuliaPy/PythonCall.jl
• https://github.com/JuliaPy/pyjuliapkg

These recipes were generated by grayskull and modified to add the py prefix.

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I wanted to let you know that I linted all conda-recipes in your PR (recipes/pyjuliacall, recipes/pyjuliapkg) and found some lint.

Here's what I've got...

For recipes/pyjuliapkg:

• The home item is expected in the about section.
• noarch: python recipes are required to have a lower bound on the python version. Typically this means putting python >=3.6 in both host and run but you should check upstream for the package's Python compatibility.

[mkitti]

@MilesCranmer , I used the following commands to create this recipe, followed by some manual editing.

mamba create -n grayskull -c conda-forge grayskull ipython
mamba activate grayskull
grayskull pypi juliacall
grayskull pypi juliapkg

xref: JuliaPy/PythonCall.jl#378

[conda-forge-webservices]

Hi! This is the friendly automated conda-forge-linting service.

I just wanted to let you know that I linted all conda-recipes in your PR (recipes/pyjuliacall, recipes/pyjuliapkg) and found it was in an excellent condition.

[MilesCranmer]

Looks good! So my understanding is Julia will be installed in the user's home directory (i.e., not the conda env) upon first import?

(I think this sort of thing is completely fine since you can specify the julia version within the juliapkg.json, right? This is probably significantly easier than the combination of approaches we were doing earlier.)

[mkitti]

I'm still sorting out the details.

[mkitti]

Interpreting the source code, it looks like it juliacall looks for an existing Julia installation before trying to download a new one.

https://github.com/JuliaPy/pyjuliapkg/blob/32c3dfca8f4d5ec193cc1ca4e2af33f62f746559/src/juliapkg/find_julia.py#L18C5-L96

If one had the julia conda-forge package installed, it should locate that and try to use it. We'll have to do some more testing. There's also the possibility of depending on the julia conda-forge package.

[mkitti]

@MilesCranmer your job here is to keep prodding me until this goes through.

Also checking into https://gitter.im/conda-forge/conda-forge.github.io might also help.

[MilesCranmer]

In retrospect I do think it’s better to let juliapkg (via juliaup) manage julia rather than conda. Then (1) we can automatically support M1/M2 macOS and Windows, (2) juliapkg lets you specify Julia version requirements in the json file (including multiple Julia versions), so it could cause conflicts to also specify it in environment.yml, (3) it’s a much easier maintenance burden on ourselves :)

[MilesCranmer]

I also think that we’ll encounter some issues anyways, so better to just get the simplest viable solution set up, and we can pursue a more complex strategy later if needed. Rather than start complicated, find it doesn’t work, and roll back to the current simpler approach.

[MilesCranmer]

Can try importing and using it here

[mkitti]

Could you elaborate?

[MilesCranmer]

Something like this:

Suggested change

	- pip check
	- pip check
	- python -c 'from juliacall import Main as jl; assert jl.seval("1 + 1") == 2'

just to start with. We can always expand later

[MilesCranmer]

Ping @mkitti, let me know how it's going

[mkitti]

I pinged https://matrix.to/#/#conda-forge:matrix.org

[cjdoris]

Interpreting the source code, it looks like it juliacall looks for an existing Julia installation before trying to download a new one.

https://github.com/JuliaPy/pyjuliapkg/blob/32c3dfca8f4d5ec193cc1ca4e2af33f62f746559/src/juliapkg/find_julia.py#L18C5-L96

If one had the julia conda-forge package installed, it should locate that and try to use it. We'll have to do some more testing. There's also the possibility of depending on the julia conda-forge package.

It's also the documented behaviour :)

In retrospect I do think it’s better to let juliapkg (via juliaup) manage julia rather than conda. Then (1) we can automatically support M1/M2 macOS and Windows, (2) juliapkg lets you specify Julia version requirements in the json file (including multiple Julia versions), so it could cause conflicts to also specify it in environment.yml, (3) it’s a much easier maintenance burden on ourselves :)

Agreed. I like the idea of keeping the installation of julia decoupled from juliacall - the user can always opt in to installing julia with conda.

[mkitti]

One problem I foresee if someone does conda install pyjuliacall julia I'm not sure if we can guarantee that julia would be installed before pyjuliacall. I think we may need to consider a configuration check on conda environment activation.

[MilesCranmer]

One problem I foresee if someone does conda install pyjuliacall julia I'm not sure if we can guarantee that julia would be installed before pyjuliacall. I think we may need to consider a configuration check on conda environment activation.

I think even in the presence of a conda version of Julia, we can basically assume that pyjuliapkg will install another version of Julia. Especially because it wants to handle multiple versions of Julia, it just seems like it’s much safer to do it with juliaup

[mkitti]

You are starting to make a case for this not to be in conda-forge.

[MilesCranmer]

How so? It could depend on juliaup on conda-forge. But not julia for the reasons stated previously (no cross-compilation available; no windows support; no version multiplexing)

[MilesCranmer]

I guess the existing juliaup on conda-forge also doesn’t interact with julia on conda-forge, right? So it would be a similar scenario here

[mkitti]

I guess the existing juliaup on conda-forge also doesn’t interact with julia on conda-forge, right? So it would be a similar scenario here

Right, but julia is not a dependency of juliaup. If anything, perhaps juliaup should be a dependency of julia but we would be effectively vendoring or doing blatant binary repackaging in that case. That's very much not in the spirit of the conda-forge guidlines:
https://conda-forge.org/docs/orga/guidelines.html

julia probably should be dependency of this package, and I would fully expect that if I installed julia from conda-forge that the version I installed would be used. conda-forge is not just any conda channel. It's the channel from where I would expect the recipes to be built from source.

If I wanted to just download binaries from wherever and do whatever I wanted I could upload the packages to my own conda channel as follows.

https://anaconda.org/julialang/pyjuliacall
https://anaconda.org/julialang/pyjuliapkg

I probably should convert that into an organization. The packages there are generated by running build-locally.py at b25cc53 and then uploading the resulting packages.

[cjdoris]

One problem I foresee if someone does conda install pyjuliacall julia I'm not sure if we can guarantee that julia would be installed before pyjuliacall. I think we may need to consider a configuration check on conda environment activation.

I don't think installation order matters does it? JuliaCall doesn't find/install Julia until it is imported.

[cjdoris]

julia probably should be dependency of this package, and I would fully expect that if I installed julia from conda-forge that the version I installed would be used.

I believe you can set "compatibilities" in Conda packages instead of "dependencies", which I think are like weak dependencies in Julia, namely it defines a compat bound on the package but doesn't install it unless it's actually a dependency of something else. So if packages at least set a compatibility on Julia correctly, then JuliaCall should pick up the conda-forge version.

[cjdoris]

But I agree that it would more be in the spirit of conda-forge to make Julia an explicit dependency of JuliaPkg. Does the Julia package in conda-forge set PYTHON_JULIAPKG_EXE?

[cjdoris]

On further reflection, making Julia an explicit dependency makes sense and should be safe. JuliaPkg will only install Julia if neither Julia nor JuliaUp is already installed. If Julia is installed but incompatible, it raises an error. So if Julia is installed in your Conda env it will either be used or raise an error - indicating some package needs to change it's compat bounds on Julia.

I think for extra safety I'll change JuliaPkg to also raise an error if Julia is not installed but you are in a Conda environment - since then you almost definitely don't want it automatically installed.

Do you have a link to the Julia recipe please?

[MilesCranmer]

https://github.com/conda-forge/julia-feedstock/blob/main/recipe/meta.yaml

My main concern with making it a hard dependency (if that's what you mean) is that you would then not be able to install pyjuliacall on Windows or macOS M1/M2, despite these being Tier 1 supported systems. So I am a fan of just letting JuliaUp manage the install.

As to why it doesn't support those:

1. Windows seems just really hard to get working for various reasons: Initial Windows build attempt julia-feedstock#168
2. macOS M1/M2 is not supported because you can't cross-compile Julia (at all), and conda-forge only has Intel runners for compiling stuff. So we would have to build each version ourselves and upload it which I don't think anybody has time to do.

Not to mention that maintaining julia-feedstock is just a lot of additional maintenance work. Even 1.10 is not yet working on any OS: conda-forge/julia-feedstock#267.

A working "good" solution > broken perfect solution :)

[cjdoris]

A working "good" solution > broken perfect solution :)

Indeed! Those are good points.

You can have platform dependent dependencies though, so Julia could be a hard dependency on supported platforms?

Instead of building Julia yourself, why not just bundle the official binaries? Are you linking against other Conda packages?

Edit: you should probably only depend on a package if you can assure it will stay supported - if the Julia feedstock is hard to maintain, making it a soft dependency would make sense.

[MilesCranmer]

You can have platform dependent dependencies though, so Julia could be a hard dependency on supported platforms?

Hm, maybe. I worry it could lead to platform-specific bugs which would be hard to diagnose unless a maintainer owns the right architecture, compared to a uniform installation solution.

Instead of building Julia yourself, why not just bundle the official binaries? Are you linking against other Conda packages?

I think this could be done by depending on juliaup's feedstock: https://github.com/conda-forge/juliaup-feedstock. There were some discussions about making the Julia feedstock just use juliaup (conda-forge/julia-feedstock#256), but it hasn't progressed further.

Edit: you should probably only depend on a package if you can assure it will stay supported - if the Julia feedstock is hard to maintain, making it a soft dependency would make sense.

👍

[mkitti]

I'm proposing we have two packages.

Within the conda-forge channel, we depend on the julia conda-forge package.

Within the julialang channel, we do not depend on Julia.

[MilesCranmer]

I don’t think you can have a dependency on an external (out of conda-forge) dependency within conda-forge, meaning that everybody would need to maintain one package on the JuliaLang channel, and one on the conda-forge channel. Just to get compatibility on macOS ARM and Windows. It just sounds like a huge amount of work to avoid a slightly imperfect solution.

Also, I view the best part about conda-forge is to have a hermetic (https://bazel.build/basics/hermeticity) build, but multiple channels breaks this.

thoughts?

[mkitti]

As far as I can tell, conda dependencies are usually not qualified based on their channel. If you depend on "pyjuliacall" and the user installed "pyjuliacall" from the "julialang" channel, then that dependency should be satisfied.

We could similarly create "julia" and "juliaup" packages in the "julialang" channel that would default to downloading the official JuliaLang.org binaries. Meanwhile the conda-forge versions default to using the conda-forge built julia package

Essentially, we are putting the choice into the hands of the users. It's their choice if they want to use the julialang channel or not and at what priority.

[mkitti]

@ngam do you have any thoughts about what we should do here and how to move this forward?

[ngam]

With current rules, you must not depend on non-conda-forge dependencies, so within each feedstock, the only channel you can use is conda-forge. (In the past, things were different and they may change again.)

For the sake of simplicity and all issues that have been stalling the Julia effort within conda-forge, I think the most feasible solution is to make these packages decoupled from the Julia we have in conda-forge. So, I wouldn't depend on Julia here explicitly. As this is the case right now, I am approving the PR. Remember we could iterate and experiment at length once these packages are in their respective feedstocks.

The conda-forge ecosystem for Julia is still in its infancy and I am not sure if it will take off anytime soon, so why delay with complicated design decisions at this junction? We can revisit. One thing we can do is "advertise" the availability of the conda-forge Julia prominently upon installation, but still let the user actively decide if they want to add Julia from conda-forge or get it from somewhere else (juliaup, etc.).

[ngam]

Tangential, do you know who's leading the julialang channel on anaconda.org?

[ngam]

(I am asking because ... it could be easier to experiment with something like that than experiment here ... Github Actions finally support Apple Silicon, and likely Azure Pipelines will follow suit very soon, so the specific MacOS bottleneck should be eased soon)

[mkitti]

Tangential, do you know who's leading the julialang channel on anaconda.org?

That would be me.

[MilesCranmer]

Ping @mkitti, can this be merged? We can iterate in the feedstock after.

[mkitti]

Ask @ngam or https://matrix.to/#/#conda-forge:matrix.org. I'm not sure what the hold up is.

[mkitti]

Have you tried conda install julialang::pyjuliacall?

[ngam]

Ask @ngam or https://matrix.to/#/#conda-forge:matrix.org. I'm not sure what the hold up is.

I cannot merge; once ready, we should ping @conda-forge/staged-recipes to review/merge it (using the chat helps in getting that done faster)

[github-actions]

To help direct your pull request to the best reviewers, please mention a topic-specifc team if your recipe matches any of the following: conda-forge/help-c-cpp, conda-forge/help-cdts, conda-forge/help-go, conda-forge/help-java, conda-forge/help-julia, conda-forge/help-nodejs, conda-forge/help-perl, conda-forge/help-python, conda-forge/help-python-c, conda-forge/help-r, conda-forge/help-ruby,or conda-forge/help-rust. Thanks!

[ngam]

Thanks @ocefpaf ❤️