Auto patch 3rd party imports to mitigate dependency hell

[mzaks]

The Wikipedia article dependency hell provides a good definition and list of dependency hell forms. The proposed solution should solve all the forms of the dependency hell, but for the sake of clarity, we shall consider only the "Conflicting dependencies" form.

Example of conflicting dependencies

Imagine we have an application where we need to import third party library_a and library_b. However library_b also depends on library_a in version 1.7, but in our own code we depend on library_a in version 3.0 which is not backwards compatible with version 1.7. This poses an unsolvable problem in most programming languages, as they do not allow "Side-by-side installation of multiple versions" (although there are frameworks like OSGI which allow solving this kind of issue on a higher level).

Proposed solution for side-by-side installation in Mojo

In order to solve dependency hell, a future Mojo package manager could allow multi version side-by-side installation of the same package, this however is only one part of the solution.

Automatic import rewrite

Say application code and dependency definitions always take precedence and the package manager installs library_a in version 3.0 and additionally introduces library_a_1_7 which reflects the 1.7 version of the library_a, needed by library_b. As the code in library_b imports symbols form library_a with statements similar to: from library_a import *, library_b would link to library_a in version 3.0, which can potentially lead to compilation or even runtime errors. Hence I would suggest that the compiler rewrites the import statements in library_b to be from library_a_1_7 import *. Given that Mojo packages contain not only binary, but also the IR code for recompilation, this should be feasible.

[gabrieldemarmiesse]

Related: https://discord.com/channels/1087530497313357884/1161291847734403072/1161348656239231247

Rust allows this: https://stackoverflow.com/questions/58739075/how-do-i-import-multiple-versions-of-the-same-crate

I'm very interested in being able to vendor libraries easily with the mojo package manager. If I use a lib and no functions/structs of this lib is exposed publicly in my interface, I don't see why the user should have to care about what's the version of my lib and how it fits with other libraries's requirements.

e.g. I can vendor numpy if I only use numpy to do some computation and never use numpy arrays in my public interface, it improves the packaging UX for my end users as I won't add any version requirement on their numpy installed.

Many projects actually do this manually and vendor old versions. For example Pydantic: from pydantic.v1 import BaseModel or tensorflow: tf.compat.v1. It shows the need for this in production environments is real and not just a small item on the package manager wishlist.

[mzaks]

I think one could go even further if package manager is tightly integrated with the compiler. Generally as a user of a package I pin a version only to avoid compilation errors. If the package manager can check for me which version is compatible with my usage, that would be quite handy. That said there are some time version changes which do not break API but provide behavior change, in this case the user need to explicitly pin the version, which contains behavior they expect.