feat: publish to Sonatype OSSRH by davisusanibar · Pull Request #102 · substrait-io/substrait-java

davisusanibar · 2022-11-10T01:49:18Z

This PR contains:

Feature to implement release to Sonatype OSSRH
Review ci/release/README.md file to create variables needed for deployment to Sonatype OSSRH

Second part to solve #43 , please at first time review/merge #100 that is needed to this PR

davisusanibar · 2022-11-10T02:16:09Z

Consider that after this PR is merged, then immediately the first deployment to Sonatype OSSRH will start and the first version v1.0.0 will be created.

jacques-n · 2022-11-10T03:42:59Z

Hey @davisusanibar, can you rebase?

jacques-n · 2022-11-10T03:48:43Z

Hey @davisusanibar , a couple things here. The doc at least looks like it is double releasing (to github and ossrh). Let's not do that. Additionally, we should automate the movement from stage to release. Some info here about this: https://issues.sonatype.org/browse/OSSRH-48410

Will look at the detail soon...

davisusanibar · 2022-11-10T03:57:31Z

Hey @davisusanibar , a couple things here. The doc at least looks like it is double releasing (to github and ossrh). Let's not do that. Additionally, we should automate the movement from stage to release. Some info here about this: https://issues.sonatype.org/browse/OSSRH-48410

Will look at the detail soon...

Semantic-release create a release with CHANGELOG notes and also adding source code. At the same moment we are adding Isthmus native images to the release to any user could download and use the native library, please let me know if this is ok.
Related to automate the release to maven central process:
Currently we are opening the staging environment and close that to be able to consume by end user: publishToSonatype closeSonatypeStagingRepository, and yes we could do all the steps opening the staging environment, close and release with publishToSonatype closeAndReleaseSonatypeStagingRepository.

I could recommend initially to send only to staging, after we test if the library is working and all is ok we could send another PR to publish automatically.

Please let me know how do we could proceed?

davisusanibar · 2022-11-10T04:05:29Z

Hey @davisusanibar , a couple things here. The doc at least looks like it is double releasing (to github and ossrh). Let's not do that. Additionally, we should automate the movement from stage to release. Some info here about this: https://issues.sonatype.org/browse/OSSRH-48410
Will look at the detail soon...

Semantic-release create a release with CHANGELOG notes and also adding source code. At the same moment we are adding Isthmus native images to the release to any user could download and use the native library, please let me know if this is ok.

Related to automate the release to maven central process:
Currently we are opening the staging environment and close that to be able to consume by end user: publishToSonatype closeSonatypeStagingRepository, and yes we could do all the steps opening the staging environment, close and release with publishToSonatype closeAndReleaseSonatypeStagingRepository.

I could recommend initially to send only to staging, after we test if the library is working and all is ok we could send another PR to publish automatically.

Please let me know how do we could proceed?

Ohh I just see your point, ... that sound really interesting, ...let me check about how do we could automate , thanks!

davisusanibar · 2022-11-10T23:14:01Z

Hi @jacques-n I have a doubt about automate the movement from stage to release, please could you help me to clarify that?

Consider:

Every push to master create a release process as detailed:

a.) Create the release on staging
REST POST: https://s01.oss.sonatype.org/service/local/staging/profiles/<account_id_profile>/start
Gradle Task: publishToSonatype

Request:
{
	"data": {
		"description": "io.substrait:substrait:1.2.4"
	}
}

Response:
{
	"data": {
		"stagedRepositoryId": "io.substrait-1041",
		"description": "io.substrait:substrait:1.2.4"
	}
}

Problem arise with "1041", it is a sequential number that cannot be pre-calculated in the next steps.

b.) Close the release (send the artifacts to https://s01.oss.sonatype.org/content/repositories/staging):
REST POST: https://s01.oss.sonatype.org/service/local/staging/bulk/close
Gradle Task: closeSonatypeStagingRepository

Request:
{
	"data": {
		"stagedRepositoryIds": ["io.substrait-1041"],
		"description": "io.substrait:substrait:1.2.4",
		"autoDropAfterRelease": true
	}
}

c.) Release the artifacts to maven central (to: https://s01.oss.sonatype.org/content/repositories/releases/):
REST POST: https://s01.oss.sonatype.org/service/local/staging/bulk/promote
Gradle Task: closeAndReleaseSonatypeStagingRepository

Request
{
	"data": {
		"stagedRepositoryIds": ["io.substrait-1041"],
		"description": "io.substrait:substrait:1.2.4",
		"autoDropAfterRelease": true
	}
}

In case is needed to Create / Close / Release in different step there is needed to persist information per steps OR call all the task as one.

davisusanibar · 2022-11-11T00:03:26Z

Hi @jacques-n I have a doubt about automate the movement from stage to release, please could you help me to clarify that?

Consider:

Every push to master create a release process as detailed:

a.) Create the release on staging REST POST: https://s01.oss.sonatype.org/service/local/staging/profiles/<account_id_profile>/start Gradle Task: publishToSonatype
Request:
{
	"data": {
		"description": "io.substrait:substrait:1.2.4"
	}
}

Response:
{
	"data": {
		"stagedRepositoryId": "io.substrait-1041",
		"description": "io.substrait:substrait:1.2.4"
	}
}
Problem arise with "1041", it is a sequential number that cannot be pre-calculated in the next steps.

b.) Close the release (send the artifacts to https://s01.oss.sonatype.org/content/repositories/staging): REST POST: https://s01.oss.sonatype.org/service/local/staging/bulk/close Gradle Task: closeSonatypeStagingRepository
Request:
{
	"data": {
		"stagedRepositoryIds": ["io.substrait-1041"],
		"description": "io.substrait:substrait:1.2.4",
		"autoDropAfterRelease": true
	}
}
c.) Release the artifacts to maven central (to: https://s01.oss.sonatype.org/content/repositories/releases/): REST POST: https://s01.oss.sonatype.org/service/local/staging/bulk/promote Gradle Task: closeAndReleaseSonatypeStagingRepository
Request
{
	"data": {
		"stagedRepositoryIds": ["io.substrait-1041"],
		"description": "io.substrait:substrait:1.2.4",
		"autoDropAfterRelease": true
	}
}
In case is needed to Create / Close / Release in different step there is needed to persist information per steps OR call all the task as one.

One way about how do we could release to maven central automatically will be to create a rule/job that the maximum time to verify/test artifacts on Staging is 05 days, after that a Job automatically will run a promote to maven central.

POST: https://s01.oss.sonatype.org/service/local/staging/profile_repositories

<stagingRepositories>
    <data>
        <stagingProfileRepository>
            <profileId>XXXXXXXX</profileId>
            <profileName>io.substrait</profileName>
            <profileType>repository</profileType>
            <repositoryId>io.substrait-1041</repositoryId>
            <type>closed</type>
            <policy>release</policy>
            <userId>dsusanibara</userId>
            <userAgent>gradle-nexus-publish-plugin/1.1.0</userAgent>
            <ipAddress>x.y.z.w</ipAddress>
            <repositoryURI>https://s01.oss.sonatype.org/content/repositories/io.substrait-1041</repositoryURI>
            <created>2022-11-10T11:46:31.407Z</created>
            <createdDate>Thu Nov 10 11:46:31 UTC 2022</createdDate>
            <createdTimestamp>1668080791407</createdTimestamp>
            <updated>2022-11-10T11:48:23.774Z</updated>
            <updatedDate>Thu Nov 10 11:48:23 UTC 2022</updatedDate>
            <updatedTimestamp>1668080903774</updatedTimestamp>
            <description>io.substrait:substrait:1.2.4</description>
            <provider>maven2</provider>
            <releaseRepositoryId>releases</releaseRepositoryId>
            <releaseRepositoryName>Releases</releaseRepositoryName>
            <notifications>0</notifications>
            <transitioning>false</transitioning>
        </stagingProfileRepository>
        ...
        ...
    </data>
</stagingRepositories>

davisusanibar · 2022-11-11T03:26:23Z

The doc at least looks like it is double releasing (to github and ossrh). Let's not do that.

@jacques-n Please also confirm if uploaded native images to Github release is ok or need to be deleted

With this we could download binaries from Github release and use that:

$ ./isthmus-macOS-2.0.0  -c "CREATE TABLE Persons ( firstName VARCHAR, lastName VARCHAR, zip INT )" "SELECT lastName, firstName FROM Persons WHERE zip = 90210"
{
  "extensionUris": [{
    "extensionUriAnchor": 1,
    "uri": "/functions_comparison.yaml"
  }],

westonpace

I'm not really an expert on Java releases but I gave it an attempt. I have a few questions that are probably just my inexperience :)

westonpace · 2022-11-16T22:30:43Z

-    - uses: actions/checkout@v2
-      with:
-        submodules: recursive
-    - uses: DeLaGuardo/setup-graalvm@5.0
-      with:
-        graalvm: '22.0.0.2'
-        java: 'java17'
-    - name: Setup Gradle
-      uses: gradle/gradle-build-action@v2
-    - name: Report Java Version
-      run: java -version
-    - name: Install GraalVM native image
-      run: gu install native-image
-    - name: Build with Gradle
-      run: gradle nativeImage
-    - name: Smoke Test
-      run: ./isthmus/src/test/script/smoke.sh
-        ./isthmus/src/test/script/tpch_smoke.sh
-    - name: Rename the artifact to OS-unique name
-      shell: bash
-      run: |
-        value=`mv isthmus/build/graal/isthmus isthmus/build/graal/isthmus-${{ matrix.os }}`
-    - name: Publish artifact
-      uses: actions/upload-artifact@master
-      with:
-        name: isthmus-${{ matrix.os }}
-        path: isthmus/build/graal/isthmus-${{ matrix.os }}
+      - uses: actions/checkout@v2
+        with:
+          submodules: recursive
+      - uses: DeLaGuardo/setup-graalvm@5.0
+        with:
+          graalvm: '22.0.0.2'
+          java: 'java17'
+      - name: Setup Gradle
+        uses: gradle/gradle-build-action@v2
+      - name: Report Java Version
+        run: java -version
+      - name: Install GraalVM native image
+        run: gu install native-image
+      - name: Build with Gradle
+        run: gradle nativeImage
+      - name: Smoke Test
+        run: ./isthmus/src/test/script/smoke.sh
+          ./isthmus/src/test/script/tpch_smoke.sh
+      - name: Rename the artifact to OS-unique name
+        shell: bash
+        run: |
+          value=`mv isthmus/build/graal/isthmus isthmus/build/graal/isthmus-${{ matrix.os }}`
+      - name: Publish artifact
+        uses: actions/upload-artifact@master
+        with:
+          name: isthmus-${{ matrix.os }}
+          path: isthmus/build/graal/isthmus-${{ matrix.os }}


Can we avoid whitespace-only changes? It makes review trickier.

westonpace · 2022-11-16T22:32:46Z

+      - next
+      - next-major
+      - beta
+      - alpha


Do we need all these? Maybe just include the ones we actually use? Or main and next to show what could be done?

This is only configuration. It will be good to have this already in place, and ONLY create this branch on the project when it is needed or there are reason for.

westonpace · 2022-11-16T23:26:15Z

+
+We are using Sonatype OSSRH and there are repositories for `snapshots` / `staging` / `release`:
+- If your library finalize with `1.4.3-XXXXX-SNAPSHOT` -> it is going to be deployed to Snapshots repository
+- If your library finalize with `1.4.3-XXXXX-SNAPSHOT.YYYY` -> it is going to be deployed to Staging/Releases repository


What does it mean for a library to finalize? Does this mean the destination repository is based on the name of the artifacts? Or is the the value of the version field?

Let me update that. The destination repository is based on the name of the artifacts. It will be:

We are using Sonatype OSSRH, the destination repository (snapshots / staging / release) is based on the name of the artifacts:

For library name artifacts as 1.4.3-XXXXX-SNAPSHOT -> it is going to be deployed to Snapshots repository

For library name artifacts as 1.4.3-XXXXX-SNAPSHOT.YYYY -> it is going to be deployed to Staging/Releases repository

For library name artifacts as 1.4.3-XXXXX -> it is going to be deployed to Staging/Releases repository

For library name artifacts as 1.4.3 -> it is going to be deployed to Staging/Releases repository

westonpace · 2022-11-16T23:27:22Z

+#### 2. How many days do I have to approve the artifacts on the Staging repository?
+
+Please review [OSSRH-24751](https://issues.sonatype.org/browse/OSSRH-24751)


What happens if a release is not approved in the time limit?

There is not an option to recover that version. A initial proposal is to run a Job that after 5 days on staging environment automatically deploy that to release maven central, more detal at #102 (comment)

Consider: OSSRH-86341

Open staging repositories are dropped after 3 days of inactivity.

Closed staging repositories are dropped after 21 days of inactivity.

Released staging repositories are dropped within an hour or two of release.

westonpace · 2022-11-16T23:29:48Z

+          "Create a well-defined, cross-language specification for data compute operations"
+        )
+        url.set("https://github.com/substrait-io/substrait-java")
+        properties.set(mapOf("country" to "PE", "dsusanibar.type.of" to "Java"))


What is this info used for?

Sorry, that was for my internal test, let me delete that.

westonpace · 2022-11-16T23:31:07Z

+#signing
+SIGNING_KEY_ID = 193EAE47
+SIGNING_PASSWORD = password
+SIGNING_KEY = ALDqwcArqwfsdqweqwrwr


What is this value? Is it a dummy value? Or is it a secret?

This is dummy value, real value is going to read at build time from Github secrets values.

Co-authored-by: Weston Pace <weston.pace@gmail.com>

davisusanibar · 2022-11-18T15:58:52Z

Hi @westonpace please if you could take a second chance to review it.

westonpace · 2022-11-18T21:02:51Z

One way about how do we could release to maven central automatically will be to create a rule/job that the maximum time to verify/test artifacts on Staging is 05 days, after that a Job automatically will run a promote to maven central.

This sounds good to me.

Semantic-release create a release with CHANGELOG notes and also adding source code. At the same moment we are adding Isthmus native images to the release to any user could download and use the native library, please let me know if this is ok.

Let me make sure I understand. The substrait-java .jar file (for programmatic use) will be deployed to maven central. The isthmus binary executable (for scripting / command-line use) will be deployed to github

That sounds reasonable to me.

davisusanibar · 2022-11-18T22:43:45Z

One way about how do we could release to maven central automatically will be to create a rule/job that the maximum time to verify/test artifacts on Staging is 05 days, after that a Job automatically will run a promote to maven central.

This sounds good to me.

Please if we could handle this as an additional PR to we could continue with the scope of this PR. I mean the proposal is that initially deploy the artifacts into Staging automatically and deploy manually to maven central, then, when this end-to-end is working we could automate that without problems.

If that won't be possible please let me know to I could send that feature into this PR

Semantic-release create a release with CHANGELOG notes and also adding source code. At the same moment we are adding Isthmus native images to the release to any user could download and use the native library, please let me know if this is ok.

Let me make sure I understand. The substrait-java .jar file (for programmatic use) will be deployed to maven central. The isthmus binary executable (for scripting / command-line use) will be deployed to github

That sounds reasonable to me.

It will be:

Sonatype OSSRH:

Susbtrait-java (ie: core.jar)
Isthmus-java (ie: isthmus.jar)

Github asset:

Only Isthmus binary executable (for scripting / command-line use)

westonpace · 2022-11-19T03:18:37Z

An additional PR for automated releases sounds fine to me.

davisusanibar · 2022-11-21T14:03:06Z

Hi team, please let me know when we could have a time to test the promote to Sonatype Staging Environment.

Also please share the Key (ie: 103EEE97) sending to Ubuntu server at step gpg --keyserver keyserver.ubuntu.com --send-keys 103EEE97

westonpace · 2022-11-21T18:46:47Z

Also please share the Key (ie: 103EEE97) sending to Ubuntu server at step gpg --keyserver keyserver.ubuntu.com --send-keys 103EEE97

Do you perhaps mean for us to run --recv-key?

(base) pace@pace-desktop:~$ gpg --keyserver keyserver.ubuntu.com --recv-key 0x103EEE97
gpg: key 9B445A3D103EEE97: public key "david dali susanibar arce (Key to sign Java artifacts) <dsusanibara@uni.pe>" imported
gpg: Total number processed: 1
gpg:               imported: 1

westonpace · 2022-11-21T18:58:05Z

@jacques-n I think there is probably a more fundamental issue here around signing releases that you might have more experience / opinions on.

The ASF policy mandates that all source releases be signed by the release manager (and optionally as many PMC members as want to participate) and all released artifacts be signed by a committer (preferably a PMC member).

That seems a little extreme and not easily automated. If we wanted to automate it, then it requires putting all signers' keys into some centralized secret storage and so you end up with a single point of failure anyways. In that case, we might as well have a single "substrait" key that is stored in some kind of credentials vault (a github secret might be enough for now).

davisusanibar · 2022-11-22T04:11:06Z

An additional PR for automated releases sounds fine to me.

Just send PR #104 for automation.

davisusanibar · 2022-11-22T04:13:40Z

Also please share the Key (ie: 103EEE97) sending to Ubuntu server at step gpg --keyserver keyserver.ubuntu.com --send-keys 103EEE97

Do you perhaps mean for us to run --recv-key?
(base) pace@pace-desktop:~$ gpg --keyserver keyserver.ubuntu.com --recv-key 0x103EEE97
gpg: key 9B445A3D103EEE97: public key "david dali susanibar arce (Key to sign Java artifacts) <dsusanibara@uni.pe>" imported
gpg: Total number processed: 1
gpg:               imported: 1

Request was related about: (1) create public/private key, (2) send public key to ubuntu server, (3) share with me the ID generated on step 1 to update documentation with that value.

westonpace · 2022-11-22T05:11:08Z

I'm not certain I understand. If I don't sign the release with my private key then I'm not sure sharing my public key will be useful.

davisusanibar · 2022-11-22T12:21:40Z

I'm not certain I understand. If I don't sign the release with my private key then I'm not sure sharing my public key will be useful.

In this case we are using Gradle Signing Plugin. It's configured on the project to read environment variables and use that as needed.

This is the signing task Task :signMaven-publishPublication involved on the release process:

...
> Task :initializeSonatypeStagingRepository
> Task :signMaven-publishPublication
> Task :publishMaven-publishPublicationToSonatypeRepository
> Task :publishToSonatype
> Task :closeSonatypeStagingRepository

For this reason we need to create public/private keys and then upload private key information into the Github project Secrets and also send public key to public server keys.

westonpace · 2022-11-22T18:30:08Z

Closing the loop, since some of this conversation happened externally. I have created secrets for this repository named SIGNING_KEY, SIGNING_KEY_ID, and SIGNING_PASSWORD. The public certificate has been uploaded to keyserver.ubuntu.com and the ID is BF2002D7C82E92DEA00BCFD2F4A1E652C8BA52B3. This key will expire in six months.

davisusanibar · 2022-11-22T23:33:57Z

Just to mention that is needed to decide what will be our first version to promote: v0.x.y or v1.0.0.

To be aligned with Sustrait version that recommend version with format 0.x.y is needed to create a dummy tag (ie: v0.0.0) then semantic release process will increment that version number (ie: v0.1.0).

Semantic release consider tags created by main branch, and increment that version number for next releases, if there aren't any tags then semantic release will create v1.0.0 tag and release 1.0.0 version.

@jacques-n / @westonpace

westonpace · 2022-11-23T01:12:56Z

@davisusanibar Can we use the instructions here which use a token instead of a username / password?

davisusanibar · 2022-11-23T01:55:20Z

@davisusanibar Can we use the instructions here which use a token instead of a username / password?

Yes, we are not using our user/password to login into UI dashboard, instead we are using tokens (at the end it's also a combination of a name and password codes but not linked to a end user)

westonpace · 2022-11-23T21:40:46Z

I have created the necessary secrets at the repository level (no promises they are all correct 😆 but let me know if it looks like any are not working when you try):

westonpace · 2022-11-23T21:41:56Z

To be aligned with Sustrait version that recommend version with format 0.x.y is needed to create a dummy tag (ie: v0.0.0) then semantic release process will increment that version number (ie: v0.1.0).

I agree that we should start with v0.1.0 but move quickly to v1.0.0 (e.g. don't need to wait for spec@1.0.0)

davisusanibar · 2022-11-24T14:06:35Z

I have created the necessary secrets at the repository level (no promises they are all correct 😆 but let me know if it looks like any are not working when you try):

This looks good.

How do you see to have an option to test Github Credentials? This PR #107 help on that. Could be an option to merge that before, then run workflow manually and when all credentials are validate we could continue to merge this PR, if not let's continue with this PR merge and let's see.

davisusanibar · 2022-11-24T14:11:33Z

To be aligned with Sustrait version that recommend version with format 0.x.y is needed to create a dummy tag (ie: v0.0.0) then semantic release process will increment that version number (ie: v0.1.0).

I agree that we should start with v0.1.0 but move quickly to v1.0.0 (e.g. don't need to wait for spec@1.0.0)

Ok. To start with v0.1.0 please help me to create a initial tag v.0.0.0 before to merge (Why? If not, semantic release will create a tag for us but this will have v1.0.0 format).

* refine doc on Velox backend * revert pom change

Enable publishing to Sonatype OSSRH Co-authored-by: Weston Pace <weston.pace@gmail.com>

davisusanibar added 4 commits November 2, 2022 13:30

build: configure commitlint, editorconfig and dry run release

b4de921

fix: initial

36aa4f1

feat: add publish to sonatype repository

6771f18

Merge branch 'main' into feature/release_to_sonatype

46184b9

davisusanibar mentioned this pull request Nov 10, 2022

[Java][CI]: Publish Java packages to artifactory server #98

Closed

13 tasks

davisusanibar added 2 commits November 9, 2022 23:20

build: solve merge conflicts

5ed4bc4

build: delete duplicate close sonatype

bea769b

build: solve error for native images release

8a8c650

westonpace reviewed Nov 16, 2022

View reviewed changes

davisusanibar and others added 2 commits November 18, 2022 09:23

fix: apply suggestions from code review

9e72807

Co-authored-by: Weston Pace <weston.pace@gmail.com>

fix: clean code + update documentation

57947e7

davisusanibar force-pushed the feature/release_to_sonatype branch from 2ecc7c2 to 57947e7 Compare November 18, 2022 15:51

davisusanibar changed the title ~~[CI]: Feature to implement release to Sonatype OSSRH~~ feat: publish to Sonatype OSSRH Nov 18, 2022

docs: add detail for release validation

868a410

davisusanibar mentioned this pull request Nov 22, 2022

feat: automation promote from staging to maven central #104

Closed

davisusanibar added 2 commits November 22, 2022 14:05

docs: update documentation about sign key id

dcacb9f

ci: configure commit rule to convert breaking rules to a minor

854f2e9

jacques-n merged commit 7b195b0 into substrait-io:main Nov 25, 2022

baibaichen pushed a commit to baibaichen/substrait-java that referenced this pull request Feb 6, 2023

[GJ-13] Refine docs on Velox backend (substrait-io#102)

d7cd5f9

* refine doc on Velox backend * revert pom change

ajegou pushed a commit to ajegou/substrait-java that referenced this pull request Mar 29, 2024

feat: publish to Sonatype OSSRH (substrait-io#102)

b72853c

Enable publishing to Sonatype OSSRH Co-authored-by: Weston Pace <weston.pace@gmail.com>

		#### 2. How many days do I have to approve the artifacts on the Staging repository?

		Please review [OSSRH-24751](https://issues.sonatype.org/browse/OSSRH-24751)

Conversation

davisusanibar commented Nov 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davisusanibar commented Nov 10, 2022

Uh oh!

jacques-n commented Nov 10, 2022

Uh oh!

jacques-n commented Nov 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davisusanibar commented Nov 10, 2022

Uh oh!

davisusanibar commented Nov 10, 2022

Uh oh!

davisusanibar commented Nov 10, 2022

Uh oh!

davisusanibar commented Nov 11, 2022

Uh oh!

davisusanibar commented Nov 11, 2022

Uh oh!

westonpace left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davisusanibar commented Nov 18, 2022

Uh oh!

westonpace commented Nov 18, 2022

Uh oh!

davisusanibar commented Nov 18, 2022

Uh oh!

westonpace commented Nov 19, 2022

Uh oh!

davisusanibar commented Nov 21, 2022

Uh oh!

westonpace commented Nov 21, 2022

Uh oh!

westonpace commented Nov 21, 2022

Uh oh!

davisusanibar commented Nov 22, 2022

Uh oh!

davisusanibar commented Nov 22, 2022

Uh oh!

westonpace commented Nov 22, 2022

Uh oh!

davisusanibar commented Nov 22, 2022

Uh oh!

westonpace commented Nov 22, 2022

Uh oh!

davisusanibar commented Nov 22, 2022

Uh oh!

davisusanibar commented Nov 10, 2022 •

edited

Loading

jacques-n commented Nov 10, 2022 •

edited

Loading