fix: pypi package should not pin dependencies precisely

[nhoening]

closes #668

[nhoening]

Does this require a changelog entry?

[Flix6x]

This might benefit from a simple example in which some package has two dependencies: FM, and a FM dependency. Maybe in the docs, but also handy for review (testing). Or is that something for the plugin cookie cutter instead?

[nhoening]

Who is the intended audience for that example?

I believe FM now behaves more like one expects from pypi packages.

[Flix6x]

True.

Would this PR also mean that when have only FM as a dependency, you get it with the latest of all of FM's dependencies? Some of these would be untested by us to work with FM. For example, pandas==2 would be installed when all of FM's dependencies support Pandas 2, and that would currently lead to problems in FM. We still might want to max FM dependencies to the most recent versions we have actually tested, which may be the ones from requirements.txt (but used as a max instead of as an equality).

[nhoening]

The drawback with adding max versions is that (a rather recent version of) FlexMeasures becomes unusable as soon as one little library releases a non-significant release beyond what we tested against.

I believe this is a game that cannot be played to perfection, but just needs good continuous attention.

For intance, I guess we should add something like pandas < 2 if we see that an (upcoming) major version is probably not supported. And maybe comment with an issue ID where that support is tested and discussed.

We currently have problems with major versions of major libraries with redis, pandas and SQLAlchemy.
I reecently removed Flask-Security from that list.

[nhoening]

Would this PR also mean that when have only FM as a dependency, you get it with the latest of all of FM's dependencies?

Yes, you get the latest stable versions, as per the way pip works.

[Flix6x]

The drawback with adding max versions is that (a rather recent version of) FlexMeasures becomes unusable as soon as one little library releases a non-significant release beyond what we tested against.

I don't understand how that works. Why would that make FM become unusable?

[Flix6x]

But in general, I agree. I just wonder whether we should include some sort of fallback instructions in case pip installing FM installs a broken version (due to untested dependencies). Something like pip install flexmeasures[stable] as a suggested fallback.

[nhoening]

I don't understand how that works. Why would that make FM become unusable?

Alright, it's not immediate.
Only under the condition that any other library or environment pins anything. Which we already do at the moment, so we're betting on being the only one. Bad policy which can easily backfire.
Especially once other libraries begin adding minimal version requirements for dependencies.

[nhoening]

But in general, I agree. I just wonder whether we should include some sort of fallback instructions in case pip installing FM installs a broken version (due to untested dependencies). Something like pip install flexmeasures[stable] as a suggested fallback.

The .txt requirement files exist and can be used quite directly.
Your set idea is interesting but non-standard, so that makes it less attractive to me. And I would consider a different name.

[Flix6x]

Okay I understand it better now. Just to be precise about the scope of this PR: let me know if the below is inexact.

We forfeit control over what exact dependency versions are installed, but only when FM is installed as a package, which is the expected behaviour for packages anyway.

This PR affects developers installing FM as a package. It doesn't affect developers installing the dev version, and also not hosts using FM via Docker, where we still control the exact versions used.

We are aware that first-time triers could run into run errors due to dependency issues, but this risk is mitigated by:

• Core developers regularly upgrading dependencies on their dev environments, and putting relevant constraints into requirements.in (plus releasing a patch version so it is picked up by Pypi).
• Anything else? Maybe our test pipeline also automatically uses the latest dependency versions?

Should we modify our Issue template to ask for a list of installed dependencies?

[nhoening]

This is a good description IMO and I agree with these two ideas in principle. Or, in any case, let me look into the first one in this PR.
We can also choose to add the latest requirement versions only when we merge to main.
This is only to catch problems earlier. The resulting action is case by case.
To monitor past versions (which we otherwise stop paying attention to) would be another issue.

[nhoening]

I made an issue out of the test pipeline extension idea, as I want to review and merge this PR soon. The CI pipeline is not made worse with this PR.

[Flix6x]

Geronimo!