WIP: Dummy PR to check maint-16.0.0 status #41118
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…cal extension types (#41053) ### Rationale for this change Two specifications and one implementation of canonical extension types were added and this should be documented. ### What changes are included in this PR? This represents current state of canonical extension types. ### Are these changes tested? No, docs only. ### Are there any user-facing changes? In so much as they read docs. * GitHub Issue: #40799 Authored-by: Rok Mihevc <rok@mihevc.org> Signed-off-by: Rok Mihevc <rok@mihevc.org>
…41048) ### Rationale for this change The motivation here is to address #41047. There is severe performance drawback in reading a StringArray as value array of a DictionaryArray, because of repeated and unnecessary UTF 8 string decoding. ### What changes are included in this PR? - Added a new function Materialize() to materialize the values to a list. When materialized, GetString() reads from the vector directly. - Added test coverage. ### Are these changes tested? Yes ### Are there any user-facing changes? No. This change maintains backwards compatibility on the API surface. It is up to the client application to decide whether to materialize the array and gain performance. * GitHub Issue: #41047 Authored-by: Keshuang Shen <keshen@microsoft.com> Signed-off-by: Curt Hagenlocher <curt@hagenlocher.org>
…is now empty (#40682) ### What changes are included in this PR? `_normalize_slice` now relies on `slice.indices` (https://docs.python.org/3/reference/datamodel.html#slice.indices). ### Are these changes tested? Yes. ### Are there any user-facing changes? Fixing wrong data returned in an edge case. * GitHub Issue: #40642 * GitHub Issue: #38768 Lead-authored-by: LucasG0 <guillermou.lucas@gmail.com> Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
### Rationale for this change nanoarrow now supports most types and reading IPC streams. As discussed in the issue, there is interest in adding it to the implementation status page for visibility. ### What changes are included in this PR? A column was added to the table with the appropriate values characterizing the C library implementation status. ### Are these changes tested? Not needed (docs!) ### Are there any user-facing changes? No (docs!) * GitHub Issue: #40689 Lead-authored-by: Dewey Dunnington <dewey@voltrondata.com> Co-authored-by: Dewey Dunnington <dewey@fishandwhistle.net> Signed-off-by: Dewey Dunnington <dewey@fishandwhistle.net>
…1091) ### Rationale for this change We can't use multiple top-level `env:` in workflow. GH-40949 introduced a top-level `env:` by `macros.github_header()`. It broke workflows that already have top-level `env:`. ### What changes are included in this PR? Omit top-level `env:` key and reuse the top-level `env:` key generated by `macros.github_header()` in workflows. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: #41088 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
… for Cython 2 (#41059) ### Rationale for this change `test_make_write_options_error` has been failing on Cython 2 crossbow build because in older versions of Cython the methods were "regular" C extension method had type check automatically built in. In Cython 3 that is not the case, see cython/cython#6127 and so the check for `ParquetFileFormat` was added in #40976. ### What changes are included in this PR? Checking the error raised for both messages, type check and the check for `ParquetFileFormat` added in #40976. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: #41043 Authored-by: AlenkaF <frim.alenka@gmail.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
raulcd
requested review from
assignUser,
kou and
CurtHagenlocher
as code owners
|
Thanks for opening a pull request! If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project. Then could you also rename the pull request title in the following format? or In the case of PARQUET issues on JIRA the title also supports: See also: |
github-actions
bot
added
Component: Python
Component: C#
Component: Documentation
awaiting committer review
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
|
The Clone steps: Output: |
|
I see a similar error for |
|
Yes, I think something weird is going on with the Azure jobs. |
|
Another class of errors seems to be: I verified |
|
Another class of errors appears to be a regression in cmake: Leading to errors like: |
… C Device data interface (#41101) ### Rationale for this change It is not explicit what the value of the `ArrowDeviceArray::device_id` should be when a given device type has no notion of a device identifier (e.g., there is always only one). ### What changes are included in this PR? The text was clarified to recommend a value of -1. This was the value already used by Arrow C++. ### Are these changes tested? No tests needed (documentation) ### Are there any user-facing changes? No * GitHub Issue: #40801 Authored-by: Dewey Dunnington <dewey@voltrondata.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
…or - add support for row-major (#40867) ### Rationale for this change The conversion from `RecordBatch` to `Tensor` class now exists but it doesn't support row-major `Tensor` as an output. This PR adds support for an option to construct row-major `Tensor`. ### What changes are included in this PR? This PR adds a `row_major` option in `RecordBatch::ToTensor` so that row-major `Tensor` can be constructed. The default conversion will be row-major. This for example works: ```python >>> import pyarrow as pa >>> import numpy as np >>> arr1 = [1, 2, 3, 4, 5, 6, 7, 8, 9] >>> arr2 = [10, 20, 30, 40, 50, 60, 70, 80, 90] >>> batch = pa.RecordBatch.from_arrays( ... [ ... pa.array(arr1, type=pa.uint16()), ... pa.array(arr2, type=pa.int16()), ... ... ], ["a", "b"] ... ) # Row-major >>> batch.to_tensor() <pyarrow.Tensor> type: int32 shape: (9, 2) strides: (8, 4) >>> batch.to_tensor().to_numpy().flags C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : False WRITEABLE : True ALIGNED : True WRITEBACKIFCOPY : False # Column-major >>> batch.to_tensor(row_major=False) <pyarrow.Tensor> type: int32 shape: (9, 2) strides: (4, 36) >>> batch.to_tensor(row_major=False).to_numpy().flags C_CONTIGUOUS : False F_CONTIGUOUS : True OWNDATA : False WRITEABLE : True ALIGNED : True WRITEBACKIFCOPY : False ``` ### Are these changes tested? Yes, in C++ and Python. ### Are there any user-facing changes? No. * GitHub Issue: #40866 Lead-authored-by: AlenkaF <frim.alenka@gmail.com> Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com> Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
…ker on Windows on archery (#41120) ### Rationale for this change Windows wheels are currently failing due to the change on `ARCHERY_DEBUG=1` by default. This uses `--progress` on `docker build` which is not supported on Windows. ### What changes are included in this PR? Do not use `--progress` on the Windows builds. ### Are these changes tested? Yes on CI via archery. ### Are there any user-facing changes? No * GitHub Issue: #41119 Lead-authored-by: Raúl Cumplido <raulcumplido@gmail.com> Co-authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
…InfoSelector behaviors against Azure for generic filesystem tests (#41068) ### Rationale for this change They are failing: ```text [ FAILED ] TestAzureHierarchicalNSGeneric.DeleteDir [ FAILED ] TestAzureHierarchicalNSGeneric.DeleteDirContents [ FAILED ] TestAzureHierarchicalNSGeneric.GetFileInfoSelector ``` ### What changes are included in this PR? * `DeleteDir()`: Check not a directory case * `DeleteDirContents()`: Check not a directory case * `GetFileInfoSelector()`: * Add not a directory check for input * Add support for returning metadata for directory ### Are these changes tested? Yes. ### Are there any user-facing changes? Yes. * GitHub Issue: #41034 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
### Rationale for this change See #40695 ### What changes are included in this PR? This PR does a few things: * Substrait is upgraded to the latest version * Support is added for the parameterized timestamp type (but not literals due to substrait-io/substrait#611). * Support is added for the following arrow-specific types: * fp16 * date_millis * time_seconds * time_millis * time_nanos * large_string * large_binary When adding support for the new timestamp types I also relaxed the restrictions on the time zone column. Substrait puts time zone information in the function and not the type. In other words, to print the "America/New York" value of a column of instants one would do something like `to_char(my_timestamp, "America/New York")` instead of `to_char(cast(my_timestamp, timestamp("nanos", "America/New York")`. However, the current implementation makes it impossible to produce or consume a plan with `to_char(my_timestamp, "America/New York")` because it would reject the type because it has a non-UTC time zone. With this latest change, we treat any non-empty timezone as a timezone_tz type. In addition, I have enabled conversions from "encoded types" to their unencoded representation. E.g. a type of `DICTIONARY<INT32>` will convert to `INT32`. At a logical expression / plan perspective these encodings are irrelevant. If anything, they may belong in a more physical plan representation. Should a need for them arise we can dig into it more later. However, I believe it is better to err on the side of generating "something" rather than failing in these cases. I don't consider this last change critical and can back it out if need be. ### Are these changes tested? Yes, I added new unit tests ### Are there any user-facing changes? Yes, via the Substrait conversion. These changes should be backwards compatible in that they only add functionality in places that previously reported "Not Supported". * GitHub Issue: #40695 Lead-authored-by: Weston Pace <weston.pace@gmail.com> Co-authored-by: Benjamin Kietzman <bengilgit@gmail.com> Signed-off-by: Weston Pace <weston.pace@gmail.com>
…1070) ### Rationale for this change Loading the `null_count` attribute doesn't take into account the possible value of -1, leading to a code path where the validity buffer is accessed, but which is not necessarily present in that case. ### What changes are included in this PR? Use `data->MayHaveNulls()` instead of `data->null_count.load()` ### Are these changes tested? Yes * GitHub Issue: #41016 Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Antoine Pitrou <antoine@python.org>
This comment was marked as outdated.
This comment was marked as outdated.
### Rationale for this change Since the left anti filter implementation is based on the left semi filter, and an assertion error occurs when the left semi filter rows are empty, this problem should be fixed. ### What changes are included in this PR? swiss_join.cc and hash_join_node_test.cc ### Are these changes tested? Yes ### Are there any user-facing changes? No * GitHub Issue: #41121 Lead-authored-by: light-city <455954986@qq.com> Co-authored-by: Antoine Pitrou <pitrou@free.fr> Signed-off-by: Antoine Pitrou <antoine@python.org>
) ### Rationale for this change Upstream problem conda-forge/glib-feedstock#174 has been fixed. ### What changes are included in this PR? Revert pinning. ### Are these changes tested? Yes. ### Are there any user-facing changes? No. * GitHub Issue: #41227 Authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>
…pandas` (#40897) ### Rationale for this change Avoiding using pandas internals to create Block objects ourselves, using a new API for pandas>=3 * GitHub Issue: #35081 Authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com> Signed-off-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
|
Revision: eb5f162 Submitted crossbow builds: ursacomputing/crossbow @ maint-16.0.0-nightly-tests-5
|
|
Revision: eb5f162 Submitted crossbow builds: ursacomputing/crossbow @ maint-16.0.0-nightly-packaging-2
|
|
Revision: eb5f162 Submitted crossbow builds: ursacomputing/crossbow @ maint-16.0.0-nightly-verification-1
|
raulcd
requested review from
paleolimbot,
thisisnic,
kevingurney,
domoritz,
trxcllnt and
zeroshade
as code owners
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DO NOT MERGE.
This PR is to track some crossbow jobs to validate status of maintenance branch before creating the first RC.