Implement biweekly releases for arrow-rs, parquet-rs #292

alamb · 2021-05-15T10:13:10Z

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Implement the process that will allow us to release to crates.io every 2 weeks as described in the proposal

See official vote email chain

I plan to use this ticket to track the work related to implementing this new process

High level plan / updates:

Created the active master branch
PR to update active_release to to 4.1.0 version, and create release notes (Add changelog and bump version for proposed 4.1.0 initial bi-weekly release #298 #305)
Create example release tarball and ask @andygrove if it would be uploadable to crates.io
Create proposed release tarball and send to mailing list for a vote Mailing List Thread
(if passed) Promote release tarball to official build and then deploy to crates.io
Document Process (Document and automate new release process #299)
Put automation in place to automatically cherry-pick PRs to active_release
Remove release scripts left over from apache/arrow repo

The text was updated successfully, but these errors were encountered:

alamb · 2021-05-15T10:14:01Z

@jorgecarleitao is working on changelog creation in #274

There is a packaging issue here: #212

alamb · 2021-05-15T12:35:01Z

Here is what I am thinking for the release workflow (based on the main apache release flow. The major difference is that there is no Release Candidate)

Proposal:

Update version on active_release branch with proposed release number
create tag from active_release branch with proposed release number
Make a candidate release .tar.gz file with signatures + Upload to apache distro dev dev, e.g. https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-rs-4.1.0/
Propose release of that rc on the mailing list
if approved: move tarball to release, e.g. https://dist.apache.org/repos/dist/release/arrow/apache-arrow-rs-4.1.0/
If not approved, fix whatever the problem is and make a new tag, with a new minor release (aka don't make "RC / release candidates, just skip minor versions)

We will then treat uploading to crates.io as a post official release task done from the official source

I hope to have a script for most of this process (other than the email / approval).

alamb · 2021-05-17T13:55:49Z

I need help verifying the proposed source tarball format for the Arrow Rust releases;

Specifically, can someone please:

Download the example files and ensure they can successfully validate the signatures
Ensure that the contents of this tarball could be used to publish to crates.io

Background: I have been working on the new release process for
arrow-rs (updates in [2]). The contents and changelog in this example
release tarball are from [3] and were created using the scripts /
instructions in [1].

[1] #299
[2] #292
[3] #305

Here is an example output (including Vote Email) generated by script in [1]:

Example output:


cd /Users/alamb/Software/arrow-rs/ && ./dev/release/create-tarball.sh  0.0.3
Attempting to create /Users/alamb/Software/arrow-rs/dev/dist/apache-arrow-rs-0.0.3/apache-arrow-rs-0.0.3-f3959f59a.tar.gz from tag 0.0.3
Draft email for dev@arrow.apache.org mailing list

---------------------------------------------------------
To: dev@arrow.apache.org
Subject: [VOTE][RUST] Release Apache Arrow

Hi,

I would like to propose a release of Apache Arrow Rust
Implementation, version 0.0.3.

This release candidate is based on commit: f3959f59a6119dab23818e6eef87e0d7b58c820e [1]

The proposed release tarball and signatures are hosted at [2].
The changelog is located at [3].

Please download, verify checksums and signatures, run the unit tests,
and vote on the release.

The vote will be open for at least 72 hours.

[ ] +1 Release this as Apache Arrow Rust
[ ] +0
[ ] -1 Do not release this as Apache Arrow Rust  because...

[1]: https://github.com/apache/arrow-rs/tree/f3959f59a6119dab23818e6eef87e0d7b58c820e
[2]: https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-rs-0.0.3
[3]: https://github.com/apache/arrow-rs/blob/f3959f59a6119dab23818e6eef87e0d7b58c820e/CHANGELOG.md
---------------------------------------------------------
Running rat license checker on /Users/alamb/Software/arrow-rs/dev/dist/apache-arrow-rs-0.0.3/apache-arrow-rs-0.0.3-f3959f59a.tar.gz
OK
No unapproved licenses
Signing tarball and creating checksums
Uploading to apache dist/dev to https://dist.apache.org/repos/dist/dev/arrow/apache-arrow-rs-0.0.3
Checked out revision 47764.
A         dev/dist/apache-arrow-rs-0.0.3
A  (bin)  dev/dist/apache-arrow-rs-0.0.3/apache-arrow-rs-0.0.3-f3959f59a.tar.gz
A         dev/dist/apache-arrow-rs-0.0.3/apache-arrow-rs-0.0.3-f3959f59a.tar.gz.sha512
A         dev/dist/apache-arrow-rs-0.0.3/apache-arrow-rs-0.0.3-f3959f59a.tar.gz.asc
A         dev/dist/apache-arrow-rs-0.0.3/apache-arrow-rs-0.0.3-f3959f59a.tar.gz.sha256
Adding         dev/dist/apache-arrow-rs-0.0.3
Adding  (bin)  dev/dist/apache-arrow-rs-0.0.3/apache-arrow-rs-0.0.3-f3959f59a.tar.gz
Adding         dev/dist/apache-arrow-rs-0.0.3/apache-arrow-rs-0.0.3-f3959f59a.tar.gz.asc
Adding         dev/dist/apache-arrow-rs-0.0.3/apache-arrow-rs-0.0.3-f3959f59a.tar.gz.sha256
Adding         dev/dist/apache-arrow-rs-0.0.3/apache-arrow-rs-0.0.3-f3959f59a.tar.gz.sha512
Transmitting file data ....done
Committing transaction...
Committed revision 47765.

Also in mailing list: https://lists.apache.org/thread.html/re4a67bda78f0636bec589bafc9fb502058159852480f3e61a00ae6c1%40%3Cdev.arrow.apache.org%3E

andygrove · 2021-05-17T14:04:30Z

The tarball contains the directory 0.0.3 and it would be better if that were renamed to apache-arrow-rs-0.0.3.

andygrove · 2021-05-17T14:11:10Z

I ran the following steps:

Downloaded tarball
Checked signatures
Untarred tarball (see note above suggesting a change to the directory name)
Ran cargo test
From the arrow directory, ran cargo publish --dry-run. Looked good.
I cannot test publishing other crates because they depend on the arrow crate being published but I ran some grep commands to check that there were no SNAPSHOT versions and that looked good.

So I would say that this format looks ready to go.

alamb · 2021-05-17T14:42:52Z

Thank you @andygrove

alamb · 2021-05-17T14:48:11Z

I'll wait until tomorrow for any other comments, and prepare a release candidate build and send to the mailing list

alamb · 2021-05-18T20:40:31Z

The tarball contains the directory 0.0.3 and it would be better if that were renamed to apache-arrow-rs-0.0.3.

Sorry @andygrove I just saw this comment (after I made a release candidate). I will update my script so subsequent releases are named this way

alamb · 2021-05-24T13:24:03Z

Update: we have released version 4.1.0 to crates.io 🎉

This week I will begin putting in place the code / process for the next release (4.2.0 perhaps?) so we can test the process and tools

alamb · 2021-05-24T20:28:30Z

I started figuring out how to backport individual PRs from master to active_release

Some notes:

Here is the common ancestor commit:

git merge-base  apache/master apache/active_release
c863a2c44bffa5c092a49e07910d5e9225483193

Thus, you can get the list of commits that are on master but not active release via:a

git rev-list `git merge-base  apache/master apache/active_release`..apache/master

Here is the list in pretty format:

git rev-list --pretty=oneline `git merge-base  apache/master apache/active_release`..apache/master

5295e25 Document and automate new release process (#299)
5ac771a respect offset in utf8 and list casts (#335)
4c17ac8 feature gate ipc reader/writer (#336)
b2de544 parquet: Speed up BitReader/DeltaBitPackDecoder (#325)
91ef8e9 Enable wasm32 as a target architecture for the SIMD feature (#324)
b88ef80 fix comparison of dictionaries with different values arrays (#332) (#333)
f316798 Fix undefined behavior in FFI (#323)
a25cafb Add ported Rust release verification script (#331)
71c2159 return reference from DictionaryArray::values() (#313) (#314)
dde86b9 feature gate csv functionality (#312)
f042191 fix invalid null handling in filter (#296)
e18b356 Doctests for StringArray and LargeStringArray. (#330)
087cf17 inline PrimitiveArray::value (#329)
7f37a7f Mutablebuffer::shrink_to_fit (#318)

I have started working through this list (from bottom) creating prs using the script in #339 with an initial PR #344:

ARROW_GITHUB_API_TOKEN=$ARROW_GITHUB_API_TOKEN CHECKOUT_ROOT=/tmp/arrow-rs CHERRY_PICK_SHA=7f37a7f2a119dd83c497766265707a64d9b82307 python3 dev/release/cherry-pick-pr.py

alamb · 2021-05-25T22:21:08Z

I made several more PRs: #354 #355 #356 #357 #358 #359 as an attempt to cherry pick to release

jorgecarleitao · 2021-05-26T03:37:04Z

IMO the is no need to PR these? We could just cherry-pick and merge based on whether they have the label "breaking-change" or not. Alternatively, use an inverse label, "minor-change" (i.e. if we want to be explicit over what to include vs what not to include).

alamb · 2021-05-26T10:44:48Z

@jorgecarleitao I will think about this -- the extra human control / auditing of "we explicitly wanted to include this change in the release" is important I think. However there may be too many PRs with this method 🤔

alamb · 2021-05-26T21:11:47Z

@jorgecarleitao , upon more thought I plan to keep doing PRs for the following reason:

It is a natural place to run the CI tests to make sure there are no logical conflicts
It offers a place for the original author / committers to comment and say it should/should not be backported.
It offers a way to make cleanups / fixups and approve (if needed) for non cherry pick PRs

For clean cherry-pick PRs I will plan on just merging the backport PRs when they are green in CI rather than waiting for a re-approval (given the code was already approved and we will have another "approve the whole release" vote). If the PR needs changes, we can get it approved prior to merging

jorgecarleitao · 2021-05-26T21:14:31Z

Sounds good 👍 . One idea would be to have a single PR with all the cherry-picks, so that we only have to review one PR. We could then merge then without squashing; no strong feelings, though; was just trying to avoid a bunch of PRs that were already reviewed. ^_^

alamb · 2021-05-26T21:35:47Z

👍 Let's give this PR per backport a try for a while and see how it goes

alamb · 2021-06-04T20:38:35Z

I have added some labels to help the process here: #409

alamb · 2021-07-17T15:06:43Z

This is about as done as I expect it to be. Closing this ticket 🎉

alamb added the enhancement Any new improvement worthy of a entry in the changelog label May 15, 2021

alamb self-assigned this May 15, 2021

This was referenced May 15, 2021

Remove old release scripts #293

Merged

Remove old tasks #294

Closed

alamb mentioned this issue May 16, 2021

Add changelog and bump version for proposed 4.0.1 initial bi-weekly release #298

Closed

alamb added development-process Related to development process of arrow-rs and removed enhancement Any new improvement worthy of a entry in the changelog labels May 16, 2021

This was referenced May 16, 2021

Document and automate new release process #299

Merged

Remove some old release scripts #300

Closed

Add changelog and bump version for proposed 4.1.0 initial bi-weekly release #298 #305

Merged

This was referenced May 22, 2021

Automatic cherry-pick script #339

Merged

Use published versions of arrow rather than github shas apache/datafusion#393

Closed

Use 4.x arrow-rs from crates.io rather than git sha apache/datafusion#395

Merged

BohuTANG mentioned this issue May 24, 2021

Use 4.x arrow-rs from crates.io instead of git sha datafuselabs/databend#602

Closed

This was referenced May 24, 2021

Cherry pick Mutablebuffer::shrink_to_fit to active_release #344

Merged

Cherry pick inline PrimitiveArray::value to active_release #354

Merged

alamb mentioned this issue May 27, 2021

Add changelog and bump version for proposed 4.2.0 release #375

Merged

alamb mentioned this issue Jun 4, 2021

Add labels to cherry pick scripts + writeup #409

Merged

alamb mentioned this issue Jun 10, 2021

Add changelog and bump version for proposed 4.3.0 release #444

Merged

alamb closed this as completed Jul 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement biweekly releases for arrow-rs, parquet-rs #292

Implement biweekly releases for arrow-rs, parquet-rs #292

alamb commented May 15, 2021 •

edited

Loading

alamb commented May 15, 2021

alamb commented May 15, 2021

alamb commented May 17, 2021 •

edited

Loading

andygrove commented May 17, 2021

andygrove commented May 17, 2021 •

edited

Loading

alamb commented May 17, 2021

alamb commented May 17, 2021

alamb commented May 18, 2021

alamb commented May 24, 2021

alamb commented May 24, 2021

alamb commented May 25, 2021

jorgecarleitao commented May 26, 2021

alamb commented May 26, 2021

alamb commented May 26, 2021

jorgecarleitao commented May 26, 2021

alamb commented May 26, 2021

alamb commented Jun 4, 2021

alamb commented Jul 17, 2021

Implement biweekly releases for arrow-rs, parquet-rs #292

Implement biweekly releases for arrow-rs, parquet-rs #292

Comments

alamb commented May 15, 2021 • edited Loading

alamb commented May 15, 2021

alamb commented May 15, 2021

alamb commented May 17, 2021 • edited Loading

andygrove commented May 17, 2021

andygrove commented May 17, 2021 • edited Loading

alamb commented May 17, 2021

alamb commented May 17, 2021

alamb commented May 18, 2021

alamb commented May 24, 2021

alamb commented May 24, 2021

alamb commented May 25, 2021

jorgecarleitao commented May 26, 2021

alamb commented May 26, 2021

alamb commented May 26, 2021

jorgecarleitao commented May 26, 2021

alamb commented May 26, 2021

alamb commented Jun 4, 2021

alamb commented Jul 17, 2021

alamb commented May 15, 2021 •

edited

Loading

alamb commented May 17, 2021 •

edited

Loading

andygrove commented May 17, 2021 •

edited

Loading