GH-46609: [Release][CI] Use System GTest for macos verification #46823

raulcd · 2025-06-16T08:59:27Z

Rationale for this change

Our system GoogleTest (installed by Homebrew) and the bundled GoogleTest are mixed and we get some CI failures.

What changes are included in this PR?

Use the system GoogleTest on our verification CI jobs for macOS.
The PR also contains a minor fix on gendata due to a small bug on a Python version.

Are these changes tested?

Yes, via archery.

Are there any user-facing changes?

No

GitHub Issue: [CI][Release] Release verification jobs for macOS AMD64 fail to build GMock #46609

github-actions · 2025-06-16T08:59:51Z

⚠️ GitHub issue #46609 has been automatically assigned in GitHub to PR creator.

raulcd · 2025-06-16T09:00:22Z

@github-actions crossbow submit verify-rc-source-cpp-macos-amd64 verify-rc-source-cpp-linux-ubuntu-24.04-amd64

github-actions · 2025-06-16T09:02:34Z

Revision: a295e17

Submitted crossbow builds: ursacomputing/crossbow @ actions-a736b5ee65

Task	Status
verify-rc-source-cpp-linux-ubuntu-24.04-amd64
verify-rc-source-cpp-macos-amd64

raulcd · 2025-06-16T09:55:04Z

@github-actions crossbow submit verify-rc-source-cpp-macos-amd64 verify-rc-source-cpp-linux-ubuntu-24.04-amd64

github-actions · 2025-06-16T09:58:15Z

Revision: e2d6347

Submitted crossbow builds: ursacomputing/crossbow @ actions-d4e7beb963

Task	Status
verify-rc-source-cpp-linux-ubuntu-24.04-amd64
verify-rc-source-cpp-macos-amd64

Use GitHub syntax instead

raulcd · 2025-06-16T10:09:10Z

@github-actions crossbow submit verify-rc-source-cpp-macos-amd64 verify-rc-source-cpp-linux-ubuntu-24.04-amd64

github-actions · 2025-06-16T10:11:40Z

Revision: da3e1b0

Submitted crossbow builds: ursacomputing/crossbow @ actions-868e9b032d

Task	Status
verify-rc-source-cpp-linux-ubuntu-24.04-amd64
verify-rc-source-cpp-macos-amd64

raulcd · 2025-06-16T10:31:43Z

@github-actions crossbow submit verify-rc-source--macos-

github-actions · 2025-06-16T10:34:16Z

Revision: da3e1b0

Submitted crossbow builds: ursacomputing/crossbow @ actions-01e35ef461

Task	Status
verify-rc-source-cpp-macos-amd64
verify-rc-source-cpp-macos-arm64
verify-rc-source-cpp-macos-conda-amd64
verify-rc-source-csharp-macos-amd64
verify-rc-source-csharp-macos-arm64
verify-rc-source-integration-macos-amd64
verify-rc-source-integration-macos-arm64
verify-rc-source-integration-macos-conda-amd64
verify-rc-source-python-macos-amd64
verify-rc-source-python-macos-arm64
verify-rc-source-python-macos-conda-amd64
verify-rc-source-ruby-macos-amd64
verify-rc-source-ruby-macos-arm64

raulcd · 2025-06-16T11:13:39Z

The previous failure (Gtest build) is fixed but the integration jobs seem to be failing to generate the integration data, see:

 Successfully installed archery-0.1.0 cffi-1.17.1 click-8.2.1 pycparser-2.22
Traceback (most recent call last):
  File "/var/folders/y6/nj790rtn62lfktb1sh__79hc0000gn/T/arrow-HEAD.XXXXX.jinLaDxeh3/venv-source/bin/archery", line 8, in <module>
    sys.exit(archery())
             ~~~~~~~^^
  File "/private/var/folders/y6/nj790rtn62lfktb1sh__79hc0000gn/T/arrow-HEAD.XXXXX.jinLaDxeh3/venv-source/lib/python3.13/site-packages/click/core.py", line 1442, in __call__
    return self.main(*args, **kwargs)
           ~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/private/var/folders/y6/nj790rtn62lfktb1sh__79hc0000gn/T/arrow-HEAD.XXXXX.jinLaDxeh3/venv-source/lib/python3.13/site-packages/click/core.py", line 1363, in main
    rv = self.invoke(ctx)
  File "/private/var/folders/y6/nj790rtn62lfktb1sh__79hc0000gn/T/arrow-HEAD.XXXXX.jinLaDxeh3/venv-source/lib/python3.13/site-packages/click/core.py", line 1830, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/private/var/folders/y6/nj790rtn62lfktb1sh__79hc0000gn/T/arrow-HEAD.XXXXX.jinLaDxeh3/venv-source/lib/python3.13/site-packages/click/core.py", line 1226, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/private/var/folders/y6/nj790rtn62lfktb1sh__79hc0000gn/T/arrow-HEAD.XXXXX.jinLaDxeh3/venv-source/lib/python3.13/site-packages/click/core.py", line 794, in invoke
    return callback(*args, **kwargs)
  File "/Users/runner/work/crossbow/crossbow/arrow/dev/archery/archery/cli.py", line 873, in integration
    run_all_tests(**args)
    ~~~~~~~~~~~~~^^^^^^^^
  File "/Users/runner/work/crossbow/crossbow/arrow/dev/archery/archery/integration/runner.py", line 639, in run_all_tests
    generated_json_files = datagen.get_generated_json_files(tempdir=tempdir)
  File "/Users/runner/work/crossbow/crossbow/arrow/dev/archery/archery/integration/datagen.py", line 1890, in get_generated_json_files
    generate_primitive_case([17, 20], name='primitive'),
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/work/crossbow/crossbow/arrow/dev/archery/archery/integration/datagen.py", line 1557, in generate_primitive_case
    return _generate_file(name, fields, batch_sizes)
  File "/Users/runner/work/crossbow/crossbow/arrow/dev/archery/archery/integration/datagen.py", line 1497, in _generate_file
    col = field.generate_column(size)
  File "/Users/runner/work/crossbow/crossbow/arrow/dev/archery/archery/integration/datagen.py", line 614, in generate_column
    values.append(random_bytes(nbytes))
                  ~~~~~~~~~~~~^^^^^^^^
  File "/Users/runner/work/crossbow/crossbow/arrow/dev/archery/archery/integration/util.py", line 112, in random_bytes
    return random.getrandbits(nbytes * 8).to_bytes(nbytes,
           ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
TypeError: an integer is required

I am unsure why and I can't seem to find any related commit about integration tests apart from moving the JS integration tests to use the remote repository. @kou do you know if something has changed about the data generation that could cause this failure?

kou · 2025-06-17T01:21:49Z

Hmm. It seems that nbytes is np.int32 here. How about the following?

diff --git a/dev/archery/archery/integration/datagen.py b/dev/archery/archery/integration/datagen.py
index c8cc36ddc5..18f1d4803d 100644
--- a/dev/archery/archery/integration/datagen.py
+++ b/dev/archery/archery/integration/datagen.py
@@ -609,9 +609,9 @@ class BinaryField(PrimitiveField):
 
         sizes = self._random_sizes(size)
 
-        for i, nbytes in enumerate(sizes):
+        for i, np_nbytes in enumerate(sizes):
             if is_valid[i]:
-                values.append(random_bytes(nbytes))
+                values.append(random_bytes(int(np_nbytes)))
             else:
                 values.append(b"")

kou · 2025-06-17T01:30:46Z

Ah, this may be related to Python version:

Python 3.12:

import numpy as np
list(enumerate(np.random.exponential(scale=4, size=10).astype(np.int32)))
# => [(0, 0), (1, 0), (2, 4), (3, 7), (4, 1), (5, 2), (6, 7), (7, 7), (8, 1), (9, 0)]

Python 3.13:

import numpy as np
list(enumerate(np.random.exponential(scale=4, size=10).astype(np.int32)))
# => [(0, np.int32(0)), (1, np.int32(7)), (2, np.int32(5)), (3, np.int32(1)), (4, np.int32(1)), (5, np.int32(4)), (6, np.int32(0)), (7, np.int32(4)), (8, np.int32(3)), (9, np.int32(1))]

raulcd · 2025-06-17T07:59:08Z

@github-actions crossbow submit verify-rc-source-integration-*

github-actions · 2025-06-17T08:01:30Z

Revision: a1e44fc

Submitted crossbow builds: ursacomputing/crossbow @ actions-7f6847a981

Task	Status
verify-rc-source-integration-linux-almalinux-8-amd64
verify-rc-source-integration-linux-conda-latest-amd64
verify-rc-source-integration-linux-ubuntu-22.04-amd64
verify-rc-source-integration-linux-ubuntu-24.04-amd64
verify-rc-source-integration-macos-amd64
verify-rc-source-integration-macos-arm64
verify-rc-source-integration-macos-conda-amd64

raulcd · 2025-06-17T08:13:09Z

oh! This seems to be a bug on Python 3.13.4 that is getting fixed on Python 3.13.5, see:

regression in 3.13.4 Random.getrandbits when an int like object is passed python/cpython#135326

raulcd · 2025-06-17T09:50:25Z

@kou this should be ready for review now.

kou

+1

kou · 2025-06-17T21:34:34Z

dev/archery/archery/integration/datagen.py

            if is_valid[i]:
-                values.append(random_bytes(nbytes))
+                values.append(random_bytes(int(np_nbytes)))


Could you add a comment why we need this workaround and when we can remove this workaround?

I've added a comment, let me know if that's ok.

conbench-apache-arrow · 2025-06-18T16:48:43Z

After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit 8f30636.

There were 119 benchmark results with an error:

Commit Run on arm64-t4g-2xlarge-linux at 2025-06-18 08:57:30Z
- tpch (R) with engine=arrow, format=parquet, language=R, memory_map=False, query_id=TPCH-07, scale_factor=1
- tpch (R) with engine=arrow, format=native, language=R, memory_map=False, query_id=TPCH-09, scale_factor=1
and 117 more (see the report linked below)

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.

apacheGH-46609: [Release][CI] Use System GTest for macos verification

a295e17

github-actions bot added the awaiting committer review Awaiting committer review label Jun 16, 2025

Change approach to make it more configurable

da3e1b0

Use GitHub syntax instead

raulcd force-pushed the GH-46609 branch from e2d6347 to da3e1b0 Compare June 16, 2025 10:09

raulcd marked this pull request as ready for review June 16, 2025 11:15

raulcd requested review from assignUser, jonkeane and kou as code owners June 16, 2025 11:15

Force cast to int for np.int32 case on py13

a1e44fc

kou approved these changes Jun 17, 2025

View reviewed changes

github-actions bot added awaiting merge Awaiting merge and removed awaiting committer review Awaiting committer review labels Jun 17, 2025

Add comment on why are we casting to int

4357339

github-actions bot removed the awaiting merge Awaiting merge label Jun 18, 2025

github-actions bot added the awaiting changes Awaiting changes label Jun 18, 2025

kou merged commit 8f30636 into apache:main Jun 18, 2025
9 checks passed

kou removed the awaiting changes Awaiting changes label Jun 18, 2025

kou mentioned this pull request Jun 18, 2025

[CI][Release] Release verification jobs for macOS AMD64 fail to build GMock #46609

Closed

raulcd deleted the GH-46609 branch June 18, 2025 08:07

GH-46609: [Release][CI] Use System GTest for macos verification #46823

GH-46609: [Release][CI] Use System GTest for macos verification #46823

Uh oh!

Conversation

raulcd commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Jun 16, 2025

Uh oh!

raulcd commented Jun 16, 2025

Uh oh!

github-actions bot commented Jun 16, 2025

Uh oh!

raulcd commented Jun 16, 2025

Uh oh!

github-actions bot commented Jun 16, 2025

Uh oh!

raulcd commented Jun 16, 2025

Uh oh!

github-actions bot commented Jun 16, 2025

Uh oh!

raulcd commented Jun 16, 2025

Uh oh!

github-actions bot commented Jun 16, 2025

Uh oh!

raulcd commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kou commented Jun 17, 2025

Uh oh!

kou commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

raulcd commented Jun 17, 2025

Uh oh!

github-actions bot commented Jun 17, 2025

Uh oh!

raulcd commented Jun 17, 2025

Uh oh!

raulcd commented Jun 17, 2025

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

kou Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

raulcd Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

kou Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

conbench-apache-arrow bot commented Jun 18, 2025

Uh oh!

Uh oh!

raulcd commented Jun 16, 2025 •

edited

Loading

raulcd commented Jun 16, 2025 •

edited

Loading

kou commented Jun 17, 2025 •

edited

Loading