Release healpy 1.17.1

[zonca]

We plan to make a new healpy release, the most important change is #910 which makes matplotlib and scipy optional dependencies.
Given this change in requirements, I think it is appropriate to name the new release 1.17.0.

I also plan to update the C++ package, even if there should not be any significant changes there.

@mreineck @hivon any feedback?

[hivon]

Since this release does not seem to address an urgent need, can you wait a couple of weeks before doing it, so I have a chance to put together a Healpix release ?

[lpsinger]

Actually, the dependency and wheel size issue is urgent for us. See nasa-gcn/gcn.nasa.gov#1974.

[zonca]

@tskisner we have some packaging changes for this release, especially #905, could you test this is not breaking installs at NERSC?

[tskisner]

I downloaded the artifact wheels produced in #905 and tested install in a virtualenv. Those wheels have a bundled libgomp but with the usual random suffix appended by cibuildwheel. There do not seem to be any conflicts with toast wheels (also produced with cibuildwheel and also containing a randomly named, vendored libgomp). However, I noticed that in the healpy feedstock there is no explicit dependence on OpenMP. Is that intentional? I was expecting something like

build:
    - llvm-openmp # [osx]
    - libgomp # [linux]

but I can open a question on that repo. Maybe it is not important- I think libgomp would get found regardless on linux builds and on macos the package would simply not use OpenMP.

[lpsinger]

@tskisner, the macOS Conda packages definitely are supposed to link against OpenMP. But yes, that would be an issue with the feedstock.

[lpsinger]

@zonca, would you be willing to do an rc or pre-release ahead of the healpix-cxx update?

[zonca]

@hivon we have a problem in #917 which is blocking the release, not sure when we will be able to release, it is a matter of another 2/3 weeks is my guess. Good news is you have time to make a HEALPix release!

[lpsinger]

We should include #933 in the next release because it is needed for ABI compatibility with Numpy 1.x and 2.x.

[zonca]

ok, #917 is merged,
@lpsinger anything else or should I make a release at this stage?

[lpsinger]

Not that I can think of. Note that we'll need a release of healpix-cxx as well.

[zonca]

@mreineck @hivon can you make a release of healpix-cxx?

[mreineck]

FIne with me, I just hope I get the shared library version right this time!

@hivon, do we make a full release, or should I do an intermediate C++-only tarball?

[hivon]

I do plan to make a full release, but it may take some time to wrap it up and integrate the new healpy in it.
So if you are in a rush, make an intermediate C++-only tarball.

[mreineck]

I think we can probably have the best of both worlds. If I know the version number (I guess 3.83?) and you don't require additional changes to the C++ part, I can just try to produce the "official" C++ only tarball in advance.

[hivon]

Yes, it would be 3.83.
Thanks

[mreineck]

@lpsinger, can you check the attached file?
healpix_cxx-3.83.0.tar.gz

[lpsinger]

@lpsinger anything else or should I make a release at this stage?

One more PR: #939

[lpsinger]

@lpsinger, can you check the attached file? healpix_cxx-3.83.0.tar.gz

I am having some problems with updating the Debian package. Please hold, @zonca. I'll elaborate shortly.

[lpsinger]

See #942.

[jvavrek]

A new release would be very useful for us as well---specifically we would need the fix from #944 available on PyPI. Otherwise we have to max version pin matplotlib in half a dozen repos.

[zonca]

@lpsinger would it be worth it to release with broken MacOS builds? #942 (comment)

Then release 1.17.1 when this is fixed in cfitsio HEASARC/cfitsio#18

[lpsinger]

The MacPorts builds are not broken. The problem is that cfitsio's build system is broken for shared libraries. MacPorts has a patch that fixes it. HEASARC/cfitsio#18 is currently blocking #942. You could certainly do a release now; the only defect that I know if is that it will be statically linked against cfitsio and therefore be a much larger distribution than it ought to be.

I say go for it. I think that the cfitsio build system rewrite is going to take a while.

[zonca]

Release:

• Github: https://github.com/healpy/healpy/releases/
• PyPI: https://pypi.org/project/healpy
• conda-forge: https://anaconda.org/conda-forge/healpy
• wheels on PyPI for linux and Mac OS

@zonca @lpsinger @hivon @mreineck

[zonca]

@lpsinger the publishing to pypi action cannot find the artifacts, however they are built and downloaded.
I tried to specify the folder 27caf8d and force push the tag again but doesn't seem to work

[zonca]

https://github.com/healpy/healpy/actions/runs/9201537407/job/25357008441

[lpsinger]

@lpsinger the publishing to pypi action cannot find the artifacts, however they are built and downloaded. I tried to specify the folder 27caf8d and force push the tag again but doesn't seem to work

See #947.

I suggest just doing another release. Version numbers cost nothing.

[lpsinger]

Remember that to trigger an upload to PyPI, you need to do a GitHub release in addition to making a tag.

[zonca]

Wheels are there: https://pypi.org/project/healpy/#files

[zonca]

@hivon 1.17.1 released.

Just waiting for https://github.com/conda-forge/healpy-feedstock bot to make the update on Conda-forge as well

[lpsinger]

According to the conda bot status, the version update is failing with the following error message:

6.00 attempts - bot error (<a href="https://github.com/regro/cf-scripts/actions/runs/9262219732">bot CI job</a>): main: Traceback (most recent call last):
  File "/home/runner/work/cf-scripts/cf-scripts/cf-scripts/conda_forge_tick/auto_tick.py", line 567, in _run_migrator_on_feedstock_branch
    migrator_uid, pr_json = run(
                            ^^^^
  File "/home/runner/work/cf-scripts/cf-scripts/cf-scripts/conda_forge_tick/auto_tick.py", line 215, in run
    migration_run_data = run_migration(
                         ^^^^^^^^^^^^^^
  File "/home/runner/work/cf-scripts/cf-scripts/cf-scripts/conda_forge_tick/migration_runner.py", line 63, in run_migration
    return run_migration_containerized(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/cf-scripts/cf-scripts/cf-scripts/conda_forge_tick/migration_runner.py", line 157, in run_migration_containerized
    data = run_container_task(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/runner/work/cf-scripts/cf-scripts/cf-scripts/conda_forge_tick/utils.py", line 249, in run_container_task
    raise RuntimeError(
RuntimeError: Error running migrate-feedstock in container - error KeyError('raw_meta_yaml') raised:
cmd: ['docker',
 'run',
 '-e',
 'CF_TICK_IN_CONTAINER=true',
 '--security-opt=no-new-privileges',
 '--read-only',
 '--cap-drop=all',
 '--mount',
 'type=tmpfs,destination=/tmp,tmpfs-mode=1777,tmpfs-size=6000000000',
 '-m',
 '6000m',
 '--cpus',
 '1',
 '--ulimit',
 'nofile=1024:1024',
 '--ulimit',
 'nproc=2048:2048',
 '--rm',
 '-i',
 '--mount',
 'type=bind,source=/tmp/tmpi5gsez8_,destination=/cf_tick_dir',
 'ghcr.io/regro/conda-forge-tick:master',
 '/opt/conda/envs/cf-scripts/bin/python',
 '-u',
 '/opt/autotick-bot/docker/run_bot_task.py',
 'migrate-feedstock',
 '--feedstock-name',
 'healpy',
 '--default-branch',
 'main',
 '--existing-feedstock-node-attrs',
 '-',
 '--kwargs',
 '{\n "hash_type": "sha256"\n}',
 '--log-level',
 'info']
stdout: {'data': None,
 'default_branch': 'main',
 'error': "KeyError('raw_meta_yaml')",
 'feedstock_name': 'healpy',
 'input_kwargs': '{\n "hash_type": "sha256"\n}',
 'traceback': 'Traceback (most recent call last):\n'
              '  File "/opt/autotick-bot/docker/run_bot_task.py", line 116, in '
              '_run_bot_task\n'
              '    data = func(attrs=attrs, **kwargs)\n'
              '           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n'
              '  File "/opt/autotick-bot/docker/run_bot_task.py", line 303, in '
              '_migrate_feedstock\n'
              '    data = run_migration_local(\n'
              '           ^^^^^^^^^^^^^^^^^^^^\n'
              '  File '
              '"/opt/autotick-bot/conda_forge_tick/migration_runner.py", line '
              '235, in run_migration_local\n'
              '    migrator.run_pre_piggyback_migrations(recipe_dir, '
              'feedstock_ctx.attrs, **kwargs)\n'
              '  File "/opt/autotick-bot/conda_forge_tick/migrators/core.py", '
              'line 417, in run_pre_piggyback_migrations\n'
              '    if not mini_migrator.filter(attrs):\n'
              '           ^^^^^^^^^^^^^^^^^^^^^^^^^^^\n'
              '  File '
              '"/opt/autotick-bot/conda_forge_tick/migrators/extra_jinj2a_keys_cleanup.py", '
              'line 34, in filter\n'
              '    raw_yaml = attrs["raw_meta_yaml"]\n'
              '               ~~~~~^^^^^^^^^^^^^^^^^\n'
              '  File '
              '"/opt/autotick-bot/conda_forge_tick/lazy_json_backends.py", '
              'line 811, in __getitem__\n'
              '    return self._data[item]\n'
              '           ~~~~~~~~~~^^^^^^\n'
              "KeyError: 'raw_meta_yaml'\n"}

I have no idea what any of that means. @duncanmmacleod?

Meanwhile, I can do the update manually.

[zonca]

conda package not available yet, see conda-forge/healpy-feedstock#67

[zonca]

@lpsinger what if we leave 1.17.1 without a conda package and ship it again in the next release?

[lpsinger]

We are shipping the source code for libsharp. The problem is that we are building it as a shared library which is not being incorporated into the conda package.

As a short term workaround, I can update the conda packaging to build libsharp as a shared library and install it. I'll work on that over the coming few days.

For the slightly longer term ... it seems that the version of libsharp that we are bundling with healpix-cxx is unmaintained, and incompatible with either of the other two standalone flavors of libsharp (https://github.com/Libsharp/libsharp and https://gitlab.mpcdf.mpg.de/mtr/libsharp). Given that, I suppose that I would suggest really vendoring libsharp directly into the healpix-cxx source so that it is no longer even a separate package.

In the much longer term, I think that you, @mreineck, and I should buckle down and talk about alternatives to healpix-cxx like ducc0 etc.

It's still really a shame that the HEALPix maintainers are set on keeping development on Subversion on SourceForge and having the very customized and brittle monorepo layout. That's not the underlying problem here, but it's an additional challenge.

[mreineck]

In the much longer term, I think that you, @mreineck, and I should buckle down and talk about alternatives to healpix-cxx like ducc0 etc.

More than happy to work on that, see #717 :-)

I'm also fine with doing the migration in steps, e.g. switch the SHTs first. From healpy's perspective that would mean a temporary increase in dependencies, but that could be worth it.

[hivon]

@zonca here: Edited out the usual @hivon's useless toxic comments which are not accepted here at all. Come on.
In summary he is saying that Healpix is happy to stay with monorepo on Sourceforge.

[lpsinger]

I finished updating the Conda packaging. Can we close this issue now?

[zonca]

I finished updating the Conda packaging. Can we close this issue now?

Thanks!