[Release][Go] Verification tasks for the Release Candidate fail on Go parquet encryption tests #38345
Assignees
Labels
Milestone
Comments
|
@zeroshade @kou I am investigating but I am really confused on why this only fails if we are verifying the Release from an official release but it is passing if we run the verification tasks from a local checkout. As reported on the issue this is reproducible locally if we use: |
|
I was able to reproduce the failure locally by manually and forcibly updating the |
|
I also think that apache/parquet-testing#39 is related. |
|
How about this? diff --git a/dev/release/verify-release-candidate.sh b/dev/release/verify-release-candidate.sh
index 0c6ac075b..287c557fb 100755
--- a/dev/release/verify-release-candidate.sh
+++ b/dev/release/verify-release-candidate.sh
@@ -959,12 +959,26 @@ ensure_source_directory() {
fi
fi
- # Ensure that the testing repositories are cloned
- if [ ! -d "${ARROW_SOURCE_DIR}/testing/data" ]; then
- git clone https://github.com/apache/arrow-testing.git ${ARROW_SOURCE_DIR}/testing
+ # Ensure that the testing repositories are prepared
+ if [ ! -d ${ARROW_SOURCE_DIR}/testing/data ]; then
+ if [ -d ${SOURCE_DIR}/../../testing/data ]; then
+ cp -a ${SOURCE_DIR}/../../testing/ ${ARROW_SOURCE_DIR}/
+ else
+ git clone \
+ https://github.com/apache/arrow-testing.git \
+ ${ARROW_SOURCE_DIR}/testing
+ fi
fi
- if [ ! -d "${ARROW_SOURCE_DIR}/cpp/submodules/parquet-testing/data" ]; then
- git clone https://github.com/apache/parquet-testing.git ${ARROW_SOURCE_DIR}/cpp/submodules/parquet-testing
+ if [ ! -d ${ARROW_SOURCE_DIR}/cpp/submodules/parquet-testing/data ]; then
+ if [ -d ${SOURCE_DIR}/../../cpp/submodules/parquet-testing/data ]; then
+ cp -a \
+ ${SOURCE_DIR}/../../cpp/submodules/parquet-testing/ \
+ ${ARROW_SOURCE_DIR}/cpp/submodules/
+ else
+ git clone \
+ https://github.com/apache/parquet-testing.git \
+ ${ARROW_SOURCE_DIR}/cpp/submodules/parquet-testing
+ fi
fi
export ARROW_TEST_DATA=$ARROW_SOURCE_DIR/testing/data(I think that this is not a blocker.) |
kou
added a commit
to kou/arrow
that referenced
this issue
Oct 19, 2023
…sible We have external test data repositories, apache/arrow-testing and apache/parquet-testing. We use them as submodule. apache/arrow may not use the latest test data repositories. But our verification script always use the latest test data repositories. It may cause test failures.
|
That's definitely a viable solution, i also just put together a PR to implement the Boolean RLE encoding... |
zeroshade
added a commit
to zeroshade/arrow
that referenced
this issue
Oct 19, 2023
raulcd
pushed a commit
that referenced
this issue
Oct 20, 2023
…38362) ### Rationale for this change We have external test data repositories, apache/arrow-testing and apache/parquet-testing. We use them as submodule. apache/arrow may not use the latest test data repositories. But our verification script always use the latest test data repositories. It may cause test failures. ### What changes are included in this PR? Use local test data if they exist. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: #38345 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
JerAguilon
pushed a commit
to JerAguilon/arrow
that referenced
this issue
Oct 23, 2023
…sible (apache#38362) ### Rationale for this change We have external test data repositories, apache/arrow-testing and apache/parquet-testing. We use them as submodule. apache/arrow may not use the latest test data repositories. But our verification script always use the latest test data repositories. It may cause test failures. ### What changes are included in this PR? Use local test data if they exist. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: apache#38345 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
This was referenced Oct 25, 2023
JerAguilon
pushed a commit
to JerAguilon/arrow
that referenced
this issue
Oct 25, 2023
…sible (apache#38362) ### Rationale for this change We have external test data repositories, apache/arrow-testing and apache/parquet-testing. We use them as submodule. apache/arrow may not use the latest test data repositories. But our verification script always use the latest test data repositories. It may cause test failures. ### What changes are included in this PR? Use local test data if they exist. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: apache#38345 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
zeroshade
added a commit
that referenced
this issue
Oct 30, 2023
### Rationale for this change Looks like the parquet-testing repo files have been updated and now include boolean columns which use the RLE encoding type. This causes the Go parquet lib to fail verification tests when it pulls the most recent commits for the parquet-testing repository. So a solution for this is to actually implement the RleBoolean encoder and decoder. ### What changes are included in this PR? Adding `RleBooleanEncoder` and `RleBooleanDecoder` and updating the `parquet-testing` repo. ### Are these changes tested? Unit tests are added, and this is also tested via the `parquet-testing` golden files. * Closes: #38345 * Closes: #38462 Lead-authored-by: Matt Topol <zotthewizard@gmail.com> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
loicalleyne
pushed a commit
to loicalleyne/arrow
that referenced
this issue
Nov 13, 2023
…sible (apache#38362) ### Rationale for this change We have external test data repositories, apache/arrow-testing and apache/parquet-testing. We use them as submodule. apache/arrow may not use the latest test data repositories. But our verification script always use the latest test data repositories. It may cause test failures. ### What changes are included in this PR? Use local test data if they exist. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: apache#38345 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
loicalleyne
pushed a commit
to loicalleyne/arrow
that referenced
this issue
Nov 13, 2023
…pache#38367) ### Rationale for this change Looks like the parquet-testing repo files have been updated and now include boolean columns which use the RLE encoding type. This causes the Go parquet lib to fail verification tests when it pulls the most recent commits for the parquet-testing repository. So a solution for this is to actually implement the RleBoolean encoder and decoder. ### What changes are included in this PR? Adding `RleBooleanEncoder` and `RleBooleanDecoder` and updating the `parquet-testing` repo. ### Are these changes tested? Unit tests are added, and this is also tested via the `parquet-testing` golden files. * Closes: apache#38345 * Closes: apache#38462 Lead-authored-by: Matt Topol <zotthewizard@gmail.com> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
raulcd
pushed a commit
that referenced
this issue
Dec 6, 2023
…38362) ### Rationale for this change We have external test data repositories, apache/arrow-testing and apache/parquet-testing. We use them as submodule. apache/arrow may not use the latest test data repositories. But our verification script always use the latest test data repositories. It may cause test failures. ### What changes are included in this PR? Use local test data if they exist. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: #38345 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
dgreiss
pushed a commit
to dgreiss/arrow
that referenced
this issue
Feb 19, 2024
…sible (apache#38362) ### Rationale for this change We have external test data repositories, apache/arrow-testing and apache/parquet-testing. We use them as submodule. apache/arrow may not use the latest test data repositories. But our verification script always use the latest test data repositories. It may cause test failures. ### What changes are included in this PR? Use local test data if they exist. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * Closes: apache#38345 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
dgreiss
pushed a commit
to dgreiss/arrow
that referenced
this issue
Feb 19, 2024
…pache#38367) ### Rationale for this change Looks like the parquet-testing repo files have been updated and now include boolean columns which use the RLE encoding type. This causes the Go parquet lib to fail verification tests when it pulls the most recent commits for the parquet-testing repository. So a solution for this is to actually implement the RleBoolean encoder and decoder. ### What changes are included in this PR? Adding `RleBooleanEncoder` and `RleBooleanDecoder` and updating the `parquet-testing` repo. ### Are these changes tested? Unit tests are added, and this is also tested via the `parquet-testing` golden files. * Closes: apache#38345 * Closes: apache#38462 Lead-authored-by: Matt Topol <zotthewizard@gmail.com> Co-authored-by: Sutou Kouhei <kou@cozmixng.org> Signed-off-by: Matt Topol <zotthewizard@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug, including details regarding any error messages, version, and platform.
We have created the RC 2 for the Apache Arrow 14.0.0 release and currently we are facing the following error on the release verification for the go sources:
This is reproducible locally with:
We didn't noticed before because this doesn't fail with non-official releases (local source):
See error here: https://github.com/ursacomputing/crossbow/actions/runs/6572466559/job/17853633830
And workflow file: https://github.com/ursacomputing/crossbow/actions/runs/6572466559/workflow
Successful one on maintenance branch where we test from local: https://github.com/ursacomputing/crossbow/actions/runs/6572711355/job/17854324601
And workflow file: https://github.com/ursacomputing/crossbow/actions/runs/6572711355/workflow
Component(s)
Go, Release
The text was updated successfully, but these errors were encountered: