Prepare 4.0.0 release candidate 1

[mpenkov]

#2963

Gentlemen, could you please review the https://github.com/RaRe-Technologies/gensim/tree/4.0.0.rc1 tag and let me know if you spot anything out of the ordinary?

It looks like the wheels are building fine on CI, so we can make this release pretty much any time from now.

[piskvorky]

Not sure what to look for in the tag, but all the Milestone tickets are gone, so good to go! 🚀

[piskvorky]

I cleaned up the 4.0.0rc1 release text a bit. Also edited all PRs titles on Github, so they look consistent next time they're regenerated.

@mpenkov I see the release text is a "delta" = only changes since 4.0.0beta, right? For the full 4.0.0, IMO we include all changes since 3.8.3, not only since 4.0.0rc1. Do you agree?

Is this good to release: PyPI + website?

[mpenkov]

Yes, for the proper release, we should make the release notes complete. We should do this before the actual release, because changing them after the wheels are made isn't ideal (the wheels are made from a tag, any changes we made afterwards won't be part of the tag).

I think this is good to release. @gojomo Can you give it a once-over? If not, I'll let the cat out of the bag sometime tomorrow or Tuesday at the latest.

[piskvorky]

@mpenkov if the 4.0.0rc1 release went well (wheels, PyPI, etc), let's do the full release right after. We don't want to postpone that any more.

RC1 is just to double check the release process internally (4.0.0beta was quite tricky IIRC). I'm not even going to promote 4.0.0rc1 to users.

[mpenkov]

OK, we're good for RC1: https://pypi.org/project/gensim/4.0.0rc1/

[gojomo]

Sorry, I haven't had a chance to try exactly RC1 by either tag or via pip, and wouldn't be able to for another couple days.

@mpenkov if the 4.0.0rc1 release went well (wheels, PyPI, etc), let's do the full release right after. We don't want to postpone that any more.

RC1 is just to double check the release process internally (4.0.0beta was quite tricky IIRC). I'm not even going to promote 4.0.0rc1 to users.

The beta found enough useful things that making a formal RC1 announcement, & giving expert/attentive users at least a week to try RC1 in their environments, might also find some last-minute glitches or regressions – saving us from any quickie 4.0.1 release. Isn't the call "please take a last look, this is about to become final" almost costless, even if no-one takes the initiative in time?

(Personally, I tend to find this kind of explicit/deliberate signalling of an immenent release, & request-for-a-look from a project, confidence-inspiring. It's a reminder that anyone could help with final verification of the release, even if people very rarely do. If someone's been simmering with a concern, it might motivate them to bring it up sooner rather than later. And if something slips by, those who saw the call will feel a shared responsibility for not checking it when they could have. Like I'm feeling right now, from seeing this issue a little late - but with more time to react, and across a larger audience.)

[piskvorky]

@gojomo okay. I'll tweet & post on the mailing list.

However, I did a test today on my OSX and gensim==4.0.0rc1 from PyPI installs but then fails to work:

>>> import gensim

ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

This was on numpy 1.18.4. Numpy 1.19 fails in the same way (1.19.0 through 1.19.5).

Only numpy 1.20 (the very latest release) works. It looks like numpy 1.20 changed the C API, broke binary compatibility: numpy/numpy#16938

What options do we have? I see two:

1. We require numpy >= 1.20, and generate wheels accordingly.
2. We require numpy < 1.20, and generate wheels accordingly.

Neither option is great, a large percentage of users are fucked either way. If my understanding above is correct, this was a pretty shit move by numpy, breaking compatibility like that in a minor version (although maybe they don't use semantic versioning?).

[piskvorky]

OK, reading the numpy discussion linked above:

If there is a "simple" (albeit annoying, I am assuming that this annoyance is of an acceptable level), solution, in that all that is needed is compiling with an older NumPy version.

So what we should do, now and in general, is to build with the oldest numpy we support, which is 1.11.3, or oldest-supported-numpy. The gensim wheels should then work across all numpy versions, including 1.20+. So, not too dramatic after all.

@mpenkov are you able to add such constraint to our CI + wheel builds?

[mpenkov]

I think so. I've pushed a commit to test such a build: piskvorky/gensim-wheels@fe0b85f

The wheels will be on S3 here: http://gensim-wheels.s3-website-us-east-1.amazonaws.com/

They will overwrite the RC1 wheels, but that doesn't matter, since the wheels are already on PyPI.

[mpenkov]

It turns out things aren't so simple. The numpy wheels for 1.11.3 aren't available for the platform/pythonversion combinations that we're building gensim wheels for. So our CI has to do its own build of numpy from source, and that's a giant can of worms that we don't want to open (have a look at the build output above, if you dare).

So, instead of targeting the oldest-supported-numpy, I think we target the oldest-supported-numpy-with-wheels, where look for wheels for the Python version that gensim targets (3.6, 3.7 and 3.8). It looks like 1.17.0 doesn't have wheels for Py3.8, so that numpy version is out. 1.18.0 has them, so that's the one we should try building with, IMHO.

[piskvorky]

And what does the literal oldest-supported-numpy (linked above) do? Does that include support for wheels?

[mpenkov]

I think it uses a rolling version of numpy. So older Python versions use an older numpy version.

I'm not sure if we can actually use it directly, because its an additional package (as opposed to an actual numpy pin). I'll give it a try.

[mpenkov]

Good news @piskvorky , using the oldest-supported-numpy as build dependency worked, and the wheels got built on the first try. I don't think that's ever happened before ;)

[gojomo]

Was this numpy-incompatibility unique to OSX, and that's why the Linux/Windows builds/tests didn't trigger the same error?

[piskvorky]

I suspect the tests ran in environments with matching numpy versions (wheel built with numpy==1.20.0, tests ran with numpy==1.20.0), so this error never materialized.

[gojomo]

If we were claiming, in install specs, to have supported numpy-1.11.3 – such that an install to a developer machine with numpy-1.11.3 thought it didn't need anything else and broke on import – it seems that should've been discovered by automatic tests.

If we roll back the build/test machine to specify an older numpy, we'll miss any bugs/incompatibilities with the latest numpy.

If we only test as far back as numpy-1.18.0, should we explicitly state that's the oldest version supported?

Perhaps we need to build/test with both the odlest and newest versions we support of Numpy & similar libraries?

[piskvorky]

IIRC there was a discussion about using a more complex testing matrix. Testing against the "oldest supported version of dependencies" was definitely planned, but not sure if implemented.

[piskvorky]

@mpenkov so can we close this now, ready for "full 4.0" Wed/Thu? Any other rough edges, reasons to postpone?

4.0 is not the final dot anyway, there are post releases etc.

[mpenkov]

Yes, I think we can go for 4.0.

Did you want to announce RC1 and give people a chance to play with it for a day or two?

[piskvorky]

I already announced RC1 yesterday. "People playing" not necessary IMO; no major change since 4.0.0beta, which had been out for 4 months – more than enough time to play.

Let me clean up (merge) the changelog and we're good to go!

[piskvorky]

Actually, can you generate the 4.0.0 change log since 4.0.0rc1 please, and push it?

After that I'll edit it, so that the 4.0.0 changelog = 4.0.0beta + 4.0.0rc1 + 4.0.0 (all changes since 3.8.3).

And we can release right after.

[mpenkov]

There's only one PR merged since RC1 that's worth mentioning - I just documented it now.