Use julia_project to manage Julia dependency #100
Conversation
|
Hmm, this project still is setup for Travis. @tfk do you have a suggest CI setup to test this? |
|
You could use I think you can use these two projects together as they are, without I should probably fork and modify quickpomdps just so I can experiment with ways to get the two projects to work together. One clue might be in something David Anthoff wrote: The Project.toml (and Manifest.toml) serves two purposes, to define packages and to define environments. The uses are separate and the file EDIT: PythonCall.jl manages Julia dependencies from from python by using But, PythonCall.jl is not flexible enough. In |
There was a problem hiding this comment.
As commented in JuliaPy/pyjulia#473, I think there should be a language-agnostic way to handle julia installations with a single transparent user interface. It'd be bad if each language/framework handles Julia installations in its own way. It seems like https://github.com/JuliaLang/juliaup is the closest and most official approach to this.
That said, I'm not actively working on this right now and I don't want to block people who want to improve Julia-Python interop.
| console_logging=False | ||
| ) | ||
|
|
||
| julia_project.run() |
There was a problem hiding this comment.
I strongly suggest avoiding side-effects on import, at least from the top-level module (i.e., on import diffeqpy). It makes it impossible to provide APIs that can/should be used before initialization. Maybe it's OK to do it in import diffeqpy.de.
There was a problem hiding this comment.
makes it impossible to provide APIs that can/should be used before initialization
I don't see at the moment when this would happen. But, I suspect you are correct. So, you would have to do import diffeqpy, then diffeqpy.setup() or something like that I suppose.
There was a problem hiding this comment.
import diffeqpy, thendiffeqpy.setup()
Yes, that's the idea. It'd be better if
>>> import diffeqpy
>>> diffeqpy.update(julia=True) # hypothetical API to update DifferentialEquations and Julia
>>> from diffeqpy import deworks. If we initialize things in import diffeqpy, it'd be impossible to use updated julia and DifferentialEquations (though the latter is somewhat possible if we integrate Revise).
I also consider that it's a language-agnostic best practice to avoid "magic" initialization on module/library import.
It installs packages and builds a system image in both the source dir and the environment created for tox.
I don't think this is related to the package initialization, though. Do you generate a sysimage for each libpython? If so, it might be due to that tox (or rather virtualenv) creates a Python executable and libpython for each environment (IIRC). I remember venv to be more "light weight" than virtualenv since it doesn't copy executable and libpython (IIRC). So maybe using tox-venv could help.
There was a problem hiding this comment.
I also consider that it's a language-agnostic best practice to avoid "magic" initialization
Yes, the idea of import package reading a bunch of state from the filesystem plus implicit inputs (environment variables) and then changing a bunch of state seems like its abusing the import system. I can't see an actual serious problem, but it doesn't feel right. I really like the idea of the user not being required to remember to do anything to use the package. But, requiring setup or install the first time is perhaps not so bad.
it'd be impossible to use updated julia and DifferentialEquations
You could require that the user restart after upgrading. I think people are pretty used to software that tells them they have to do that.
Do you generate a sysimage for each libpython?
If there is a conflicting libpython then julia_project gives the user the option of either rebuilding PyCall (and breaking whatever built it previously) or using a depot particular to the project and building a new PyCall (and all other packages) there. (I don't know of a more fine-grained option at the moment) In the tox test, so far, I only choose the latter. But, I think it may not be that. I did not spend much time debugging tox so far.
There was a problem hiding this comment.
But, requiring
setuporinstallthe first time is perhaps not so bad.
I'm OK with from diffeqpy import de working out-of-the-box by doing some magic by default. I'm just suggesting to provide an option to do some operations before booting up PyJulia and DifferentialEquations. In fact, this is exactly why diffeqpy does the initialization in diffeqpy/de.py and not in diffeqpy/__init__.py.
You could require that the user restart after upgrading. I think people are pretty used to software that tells them they have to do that.
I don't think we need to copy badly designed software when there is a simple and better solution that is already implemented.
| julia_project = JuliaProject( | ||
| name="diffeqpy", | ||
| package_path=diffeqpy_path, | ||
| preferred_julia_versions = ['1.7', '1.6', 'latest'], | ||
| env_prefix = 'DIFFEQPY_', | ||
| logging_level = logging.INFO, # or logging.WARN, | ||
| console_logging=False | ||
| ) |
There was a problem hiding this comment.
What's the reason why to do this in Python? Since Chris is the maintainer of this package, it's very likely he and his contributors want to deal with Julia programs than in Python. That's why I wrote things mainly in Julia (e.g., install.jl) and put a thin wrapper of Python.
Is this because of julia binary discovery and installation?
There was a problem hiding this comment.
I understand the gist, but maybe not precisely what you are asking. Yes, JuliaProject does binary discovery and installation via another python package, find_julia, which uses jill.py. That part could be separated out, and the much of the rest could be done in Julia. I considered this. One argument for using Python is that errors and stack traces are more likely to be in Python this way. And a Python user is more likely to be able to spot an error, or make a suggestion or improvement. I'm trying, to the extent possible, to hide Julia from the end user. Admittedly, in diffeqpy, the interface to Julia is rather low level. But in other python projects, Julia is better hidden behind a Python API. I often work in environments where 0% of the people have enough interest in Julia to ever try it for anything. And 100% of the people love Python. Something like julia_project is a good way to introduce the possibility of Julia. On the other hand, I'll think more about whether more of it might be better written in Julia.
There was a problem hiding this comment.
I often work in environments where 0% of the people have enough interest in Julia to ever try it for anything. And 100% of the people love Python. Something like
julia_projectis a good way to introduce the possibility of Julia. On the other hand, I'll think more about whether more of it might be better written in Julia.
I 100% agree with this. The only reason I could get my team to agree to use diffeqpy in our project is because of the speedup relative to SciPy ODE solvers. But the users of my project would not be interested in having to work with Julia code.
There was a problem hiding this comment.
I also strongly agree with @jlapeyre , I like Julia very much but I have to force myself to use Python every time because the organization I am associated with deals only in Python, I doubt if anyone will be ready to deal with Julia while working on a project which is all in python. It will be a good idea to hide Julia altogether while appreciating all the benefits that Julia provides :)
Here are several thoughts on choosing an installer. The bottom line is that using juliaup would be much more difficult for my purposes, which is to make installing a python module that depends on Julia as easy as installing a python module that depends on a rust or c++ library.
|
|
One problem is that tox currently takes 15 minutes to run locally. It installs packages and builds a system image in both the source dir and the environment created for tox. I'm not sure, but I suspect this may be because of what @tkf mentioned. You should be able to import the module without doing any work that has side effects. |
Why not install
I'm not sure if that's the downside. You only have to validate at least one binary for each platform. You can then use the cryptographic verification for all possible Julia binaries (and possibly new
While I love Python as an excellent language for writing scripts easily, I disagree that it's a good language for creating simple-to-distribute self-contained CLI. Of course, there are various ways to create a self-contained Python application but it's not as straightforward as using a language with an AOT compiler. While I respect the effort and passion that went into jill.py and your All that said, let me note again that I'm not working on this and I have no intention to be a blocker. Chris seems to like how things are handled in R which is similar to what is suggested in this PR, IIUC. So, I think there's a good chance this gets in. |
As I said above, jill.py is a cross-platform application. Its a CLI application If someone already has python installed or is willing to install it, then jill.py is clearly a far easier solution than juliaup. @sibyjackgrove tested
I strongly suspected this from the beginning. Why else would someone use an AOT compiler for this? I looked briefly yesterday for how to package a simple-to-distribute self-contained python application. I think I saw dead projects, old ill-maintained projects. It did not look encouraging at all. So yeah, using jill.py for people who don't want, or can't, install python would be tough. Python rules the world. In the spaces I am targeting, nothing else matters. I have to be practical given my environment and very scare resources (mainly time). They have no incentive to accommodate us. I have to accommodate them. I want to maximize the probability that the Python world accepts things like this. The more Python they see, the happier they are. (Of course, there is a small minority that has a broader view). I also have to do all of this myself, including the project that I originally wanted to do. If I get time in the future to try to support juliaup, I think it would be a good idea. By the way
Well you have by far the most experience in designing things to call Julia from Python. So, it is very useful to hear your opinions. For instance, not doing work with side-effects when importing. So thanks for taking the time to weigh in! (By the way, can you explain a bit more the situations in which side-effects on import are a problem?)
Oh, I need to check that out. |
|
Rebase onto master for CI? |
Yeah, I support the idea even though the implementation is not of my taste. It'd be great to see more Julia-based packages in PyPI. Anyway, now that we have POC jill integration merged #86, I'll stop complaining about this 🙂 BTW, consider #86 as a sketch of an implementation and feel free to tweak the CI setup if you have something else based on
I'll comment on #100 (comment) to keep the conversation linear |
It's important to think about the options and defend your choice. I plan to ask the juliaup people some questions on paths and so forth to see whats possible. I'm fairly sure the combination of the CI in #86 and tox.ini will not work with julia_project without tweaking. |
jlapeyre
force-pushed
the
julia-project
branch
from
32d1e37
to
67092f2
Compare
I tried to do that. It was a bit of a mysterious process. I think what I pushed now is correct. |
|
Latest julia_project depends on latest find_julia which has a bug fix.
|
EDIT: no wait, ignore below.
|
| @@ -56,3 +56,6 @@ docs/_build/ | |||
|
|
|||
| # PyBuilder | |||
| target/ | |||
|
|
|||
| **/Manifest.toml | |||
There was a problem hiding this comment.
Does julia_project generate files inside diffeqpy directory? I don't think that's a good idea. For example, the directory may not be writable after the installation.
I see Pluto is using ~/.julia/environments/__pluto_$VERSION for its internal env. So, similarly, maybe you can generate ~/.julia/environments/__python_julia_project_$VERSION_$SLUG/ where $VERSION is the Julia version and $SLUG is a hash of the path of the current python environment (say).
There was a problem hiding this comment.
Yes it does. But, more than just Manifest.toml and Project.toml. It optionally puts a depo in ./depot in the diffeqpy directory. I also don't like it, but this was easier to start with. I agree it should be changed. Probably under ~/.julia/julia_project. Both juliaup and pythoncall claim directories there.
There was a problem hiding this comment.
I think it's very rare that you'd need a separate depot. If a Julia environment is enough, I think that's much better. It definitely helps reduce precompile time. It's different from how Python virtualenv/venv works but I think a slug-based mapping is enough for bridging the gap.
There was a problem hiding this comment.
I'd rather just use a Julia environment as well. In fact, I do use one. But the separate depot is just to work around the problem of incompatible libpython. I want to avoid having multiple python projects fighting over PyCall, always rebuilding it.... Or maybe I don't follow what you mean here.
There was a problem hiding this comment.
I don't have a great understanding of the problem with incompatible libpython. My assumption, which I am not sure of, is that it is not enough to have a separate .ji file for PyCall for each python project. Some other .ji files might need to be recompiled and be incompatible as well. Or, is it enough to use multiple depots and one of them only contains PyCall (including the compiled .ji file) ?
There was a problem hiding this comment.
But the separate depot is just to work around the problem of incompatible libpython.
I'm pretty sure the direction of JuliaPy/PyCall.jl#945 makes this hack unnecessary. It let us configure libpython for each Julia project and so PyCall.jl can be precompiled for each libpython. It'll be a game changer for how Julia packages are used from Python.
There was a problem hiding this comment.
Wow. I looked around quite a bit, but missed this. It's a big deal. I'll have to do some reading to absorb it all.
|
https://github.com/SciML/diffeqpy/runs/4811765599?check_suite_focus=true#step:5:29
I'd write something like for x in self.results.jill_julia_bin_paths.values():
return x
return ??default??? |
|
That would probably be more clear, but the logic would be slightly different. In any case, the bug is because this is macos. The directory that jill installs to is always present, but has no julia installations. I did not exercise this path till now. if not self.results.jill_julia_bin_paths:
return None
for pref in self.preferred_julia_versions:
bin_path = self.results.jill_julia_bin_paths.get(pref)
if bin_path:
return bin_path
if self._strict_preferred_julia_versions:
return None
return next(iter(self.results.jill_julia_bin_paths.values())) # Take the first one |
There is another bug fix in find_julia
|
So, I think there are still too much book keeping logic inside of diffeqpy. I think most of the stuff should go into (1) We have from from julia_project import JuliaProject
project = JuliaProject(
name="diffeqpy",
package_path=__file__,
... other things ...
)
# end of file(2) from ._julia_project import projectThis way, a user can run (3) Invoke the magic command in from . import project
project.ensure_init()Ideally, Looking at
This way, |
I anticipated this in a comment above. I had planned to tackle this later because I did not have a clear idea of what to do. As a first step, I planned to provide a way to avoid activating the
In the end, I think this is better. I did it the other way because I wanted to hide more of the I thought of using "stacked environments", but I never managed to make that work for myself, so I shied away. I imagined I might have to do something more low-level, like parsing
Isn't this problem inherent to the using stacked environments ? I mean, is this peculiar to the "automagic" approach?
You mean, if the user wants to use two python packages that depend on Julia, then activate a Julia project and add then necessary Julia packages for each? We could make something like this possible, but I would not want to require it. Then there is the question of building system images. This is important because I want to reduce latency. If you use only a single Python package that uses |
Yes, you are right. I was sloppy. The problem is inherent to how Julia itself handles
Yeah, I get that this PR is about automation. I just wanted to point out something like
This is where "no magic init" principle is useful. If all Julia-based Python packages follow this principle and then don't initialize PyJulia on import, you can create a sysimage for each combination (in principle): import diffeqpy
import makie # hypothetical
import julia_project
julia_project.compileall() # also initialize PyJulia (maybe not a good name)
from diffeqpy import de # loaded from sysimagewhere |
|
I can't afford to make something really robust at once. If I can get something that works well enough, my company (or others) might be more interested in allocating resources. But, it's probably a good idea to try to anticipate so that the interface doesn't change too quickly. No auto-init is one item to start with. I can spend some time redesigning; but I have less time for this in the near future, I did a lot of it over holidays. Your system image idea is nice. What I have currently is simple, it just uses the API that PackageCompiler offers, and it is on the packager (me, or you or Chris) to include Currently the density of Julia-based packages in use is very low, so package-package interactions are negligible. It would be great to be in a situation where were forced to deal with interactions. |
I don't understand what you are referring to here. I don't see any book keeping. All I see that could be removed is def compile_diffeqpy():
"""
Compile a system image for `diffeqpy` in the subdirectory `./sys_image/`. This
system image will be loaded the next time you import `diffeqpy`.
"""
julia_project.compile_julia_project()which I made as as an obvious convenience. I did it this way so the user does not have to know that |
Of course, it's not like everything has to be implemented in one go. But I thought the basic design (1)--(3) I commented #100 (comment) (without the future/ideal improvements I discussed) can be done with a very small effort. Essentially everything is in this PR. So, isn't it "just" removing class JuliaProject:
...
initialized = False
def ensure_init():
if not self.initialized:
self.initialized = True
self.run()
I think we just have to document that you can call Project management
------------------
You can call methods of ``diffeqpy.project`` to manage underlying Julia projects.
Notable methods are:
``diffeqpy.project.project.compile_julia_project()``
Compile a system image for `diffeqpy` in ...
``diffeqpy.project.project.update()``
Remove possible stale Manifest.toml files and compiled system image.
...
For more details, see: https://github.com/jlapeyre/julia_projectThis way, users can get some overview by typing
Per-instance docstring is tricky but I think there are various ways to do it. Maybe you can create a subclass in class JuliaProject:
def __new__(cls, *, name, **_kwargs):
class NewJuliaProject(cls):
__doc__ = f"docstring for {name}"
return object.__new__(cls)or a simpler solution is to provide a factory function def new_project(*, name, **kwargs):
class NewJuliaProject(cls):
__doc__ = f"docstring for {name}"
return NewJuliaProject(name=name, **kwargs)Maybe the |
|
So do I merge this? |
|
I'd like to make some or most of the changes that @tkf asked for first. I was doing other things, but I am now finishing organizing/opensourcing the other application of |
|
Now managed by JuliaCall. |
This PR uses julia_project and find_julia to handle installing and managing Julia and Julia packages.