[Release] Update remove old artifacts release script

[asfimport]

I just executed the remove old artifacts release script which also removed the previously created three patch releases for 6.0.2, 7.0.1, 8.0.1.

That's not desirable since those have just been released so I had to revert to an earlier revision.

cc @kou @assignUser @raulcd

Reporter: Krisztian Szucs / @kszucs
Assignee: Kouhei Sutou / @kou

_{Note: This issue was originally created as ARROW-17294. Please see the migration documentation for further details.}

[asfimport]

Kouhei Sutou / @kou:
We need to discuss our maintenance policy. For example, we maintain the last 3 major releases (9.Y.Z, 8.Y.Z and 7.Y.Z).
If we have our maintenance policy, we can determine unmaintained versions automatically.

[asfimport]

Kouhei Sutou / @kou:
"[DISCUSS] Maintenance policy" on dev@arrow.apache.org:
https://lists.apache.org/thread/7mgwr02h1f7zvghym1kljyb32s50vx1o

[h-vetinari]

So currently, there's only arrow 11 present under https://dist.apache.org/repos/dist/release/arrow anymore. Whatever the implicit/explicit support policy that's decided, I think it's a bad idea to remove the source release tarballs (we're supporting the last 4 versions in conda-forge -- 5 while we're transitioning to any new version --; we regularly need to rebuild those branches and were relying on those URLs, e.g. conda-forge/arrow-cpp-feedstock#1017 broke; I can switch to GH sources, but then what's the point of publishing tarballs at all?).

[kou]

Could you use https://www.apache.org/dyn/closer.lua?action=download&filename=arrow/arrow-11.0.0/apache-arrow-11.0.0.tar.gz style URL used in our install page instead of direct https://dist.apache.org/repos/dist/release/arrow/ URL? The .../dyn/closer.lua?... style URL is the official URL and it works for all versions. For example: https://www.apache.org/dyn/closer.lua?action=download&filename=arrow/arrow-10.0.1/apache-arrow-10.0.1.tar.gz

[h-vetinari]

Sure. Didn't know that URL style, thanks.

[h-vetinari]

I don't see how this could be introspected though (being dynamic AFAICT), and so I think I'll stay with the github tags.

[kou]

Sorry. I'm not sure what "introspected" means. Could you explain more?

FYI: Other packaging systems such as Conan, Homebrew and MSYS2 also use the dyn/closer.lua style URL:

Conan:

• https://github.com/conan-io/conan-center-index/blob/master/recipes/arrow/all/conandata.yml#L3
• See also: arrow: use the official source archive conan-io/conan-center-index#14431

Homebrew:

• https://github.com/Homebrew/homebrew-core/blob/master/Formula/apache-arrow.rb#L4

MSYS2:

• https://github.com/msys2/MINGW-packages/blob/master/mingw-w64-arrow/PKGBUILD#L41

[h-vetinari]

Sorry. I'm not sure what "introspected" means. Could you explain more?

It cannot be taken apart and looked at. Or rather, when I do, I get to https://archive.apache.org/dist/arrow/arrow-10.0.0/, which actually looks way more useful that an url that's dynamically constructed (which is harder for the bots, harder for reproducibility -- even if you promise it --, etc.).

I don't speak for the other packaging systems, but I will always prefer source urls that are as simple possible (i.e. non-magical, not dynamically constructed...), and so I'd always take https://github.com/apache/arrow/archive/refs/tags/apache-arrow-{{ version }}.tar.gz or https://archive.apache.org/dist/arrow/arrow-{{ version }}/apache-arrow-{{ version }}.tar.gz over https://www.apache.org/dyn/closer.lua?action=download&filename=arrow/arrow-11.0.0/apache-arrow-11.0.0.tar.gz

[kou]

Hmm. conan-io/conan-center-index#14431 (comment) thread may help you.

[h-vetinari]

Thanks. But it's not adding new information. Well, except that with the policy of only keeping the last 3 releases online, I've got one more argument for using the GitHub tags.

[kou]

I can't stop you from using the GitHub tags but please keep in mind that they are NOT our "official" releases.
See also: https://www.apache.org/legal/release-policy.html
Especially: https://www.apache.org/legal/release-policy.html#publication