Update dependencies 2024/04

[glatterf42]

This PR completes most open dependency updates, most notably removing the pin on pandas < 2.2.0.

Two notes:

1. It doesn't bump httpx to 0.27 yet because that introduces a DeprecationWarning in an upstream dependency. Starlette already has a release resolving this issue (v0.37), but fastpi still pins starlette to below this version. Thus, I think it's fine if we keep from updating httpx just yet.
2. I'm not happy with my current workaround for an upcoming pandas deprecation. In two places in the code, I've seen the same DeprecationWarning:

/home/fridolin/ixmp4/ixmp4/data/db/meta/repository.py:175: DeprecationWarning: DataFrameGroupBy.apply operated on the grouping columns. This behavior is deprecated, and in a future version of pandas the grouping columns will be excluded from the operation. Either pass `include_groups=False` to exclude the groupings or explicitly select the grouping columns after groupby to silence this warning.
    return df.groupby("type").apply(map_value_column)

So I started tinkering with the options, but I couldn't figure out how to successfully select the grouping columns so that apply works on them. groupby(as_index=False, ...) doesn't seem to have an effect, which might suggest that pandas is treating our apply call as a transformation in the background. Simply including apply.(include_groups=False, ...) resulted in an error that the 'type' column was missing from the dataframe. Finally, I came across this suggestion: https://stackoverflow.com/a/78100669/23037943 (see in particular the first reply). The suggested workaround copies the column we want to groupby and then group(s)by the copied column, thus leaving the original open to be passed to apply as usual. Of course, this is temporarily creating extra data columns which we don't need, so it very much feels like there should be a better solution hiding somewhere. @meksor and @danielhuppmann, please let me know if you have any idea.

Apart from this, we probably want to release 0.7.4 after merging this PR to resolve the issue @phackstock had downstream.

FYI @pmussak

[phackstock]

Thanks a lot for the quick fix @glatterf42 👍

[meksor]

Hello I removed the copied columns and put them back in each map function. From my side this looks good and can be merged! I also updated dask because there is a brand new python3.11 bug.

[danielhuppmann]

Does the fix not work with pandas 2.1? Seems very tight to depend on a package version that was released less than 2 months ago.

[glatterf42]

The initial trigger for this PR was to remove the version pin of pandas. ~2.1.2 translates to >=2.1.2, <2.2.0, but climate-processor needs pandas 2.2.0 (if I understand correctly). Updating dependencies in poetry is usually done by running something like poetry add pandas@latest, which resolves to the latest version and uses the ^ notation, where ^2.2.1 translates to >=2.2.1, <3.0.0.
In other words, what we have now is simply the default. We could manually lower the requirement to ^2.1.2, which would allow climate-processor to resolve to the newer versions, too. However, it looks like pandas is releasing v2.2.2 today, so we won't only support the latest version either way. I think it's fine to keep the dependency as it is right now.

[danielhuppmann]

I disagree, ixmp4 is now a crucial dependency of pyam and that package is used by many people, so we should be as relaxed as possible - unless there is a good reason to be strict.
See this user feedback IAMconsortium/pyam#840

[glatterf42]

Looking into this.

[glatterf42]

Okay, so I have now used uv's wonderful lowest-direct dependency resolution to arrive at a requirements.txt file with version pins for python3.10 that are compatible with our project. This process needs as input a requirements.in file, which I generated from our current pyproject.toml by replacing all version specifiers with >=. At first, I was a bit too radical in keeping only the major version; so if we required a project's 2.1.12 version, I'd keep >= 2.0 in my requirements.in. This produced problems at times and wasn't applicable to projects with versions starting with 0.. In those cases, I looked at pypi and chose a version more or less at random that is some good six months old.
Next, I created a new venv with python 3.10 and installed all versions that were pinned in the requirements.txt (see below). And the tests are successful :)
So I could now hijack this PR and poetry add all these dependencies with >= marks to expand the dependencies we support somewhat in time. For example, we can still support

• pandas 2.0.0
• dask 2023.9.0
• fastapi 0.100.0

and several others. Please let me know what you think, @danielhuppmann

requirements.txt

The versions in here pass our test suite with python 3.10 :)

[glatterf42]

@meksor Please also let me know what you think and whether we should revert some more internal dependencies to ^ again. The dev, docs, server groups come to mind as groups that might warrant more stringent dependency limits.

[danielhuppmann]

Thanks @glatterf42, this is super-helpful!

From my point of view, we should add a nightly test (running once a week or so) to alert us when a new release of some dependency package breaks something - but given that ixmp4 is now quite stable and used in production by more and more people, better to be lenient with dependencies...

[glatterf42]

@meksor please let me know what you think of the dependency changes.

[meksor]

Has something changed from my last comment 3d ago?

[glatterf42]

Yes, we now support some older versions, i.e. several dependencies are now not as strict as they used to be.

[glatterf42]

Also, I removed essentially all upper version bounds.

[meksor]

Ah I see, well if the tests pass its fine with me, but then daniels suggestion for nightly builds seems even more salient now!

[glatterf42]

But we can do that in another PR :)