Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GH-33346: [Python] DataFrame Interchange Protocol for pyarrow Table #14804

Merged
merged 49 commits into from
Jan 13, 2023
Merged

GH-33346: [Python] DataFrame Interchange Protocol for pyarrow Table #14804

merged 49 commits into from
Jan 13, 2023

Commits on Jan 11, 2023

  1. Produce a __dataframe__ object - squshed commits from apache#14613

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    ca526a7 View commit details
    Browse the repository at this point in the history

  2. Add column convert methods

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    d0ca2b1 View commit details
    Browse the repository at this point in the history

  3. Fix linter errors

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    c356cd1 View commit details
    Browse the repository at this point in the history

  4. Add from_dataframe method details

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    8fb50c5 View commit details
    Browse the repository at this point in the history

  5. Add tests for from_dataframe / pandas roundtrip

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    4fab43b View commit details
    Browse the repository at this point in the history

  6. Skip from_dataframe tests for older pandas versions

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    47ce2d6 View commit details
    Browse the repository at this point in the history

  7. Add support for LargeStringArrays in Column class

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    1c2955f View commit details
    Browse the repository at this point in the history

  8. Add test for uint and make changes to test_offset_of_sliced_array() a… 

    …nd test_large_string()
    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    0517a6d View commit details
    Browse the repository at this point in the history

  9. Prefix table metadata with pyarrow.

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    a7313fb View commit details
    Browse the repository at this point in the history

  10. Update from_dataframe method

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    9583672 View commit details
    Browse the repository at this point in the history

  11. Try to add warnings to places where copies of data are being made

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    9e8733c View commit details
    Browse the repository at this point in the history

  12. Update python/pyarrow/interchange/column.py 

    Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
    @AlenkaF @jorisvandenbossche
    AlenkaF and jorisvandenbossche committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    855ec8a View commit details
    Browse the repository at this point in the history

  13. Expose from_dataframe in interchange/__init__.py

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    beec5aa View commit details
    Browse the repository at this point in the history

  14. Add lost whitespace lines

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    b11d84e View commit details
    Browse the repository at this point in the history

  15. Revert commented categories in CategoricalDescription, column.py

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    6c9dce4 View commit details
    Browse the repository at this point in the history

  16. Add _dtype attribute to __inti__ of the Column class and move all the… 

    … dtype logic into _dtype_from_arrowdtype helper
    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    8d91b67 View commit details
    Browse the repository at this point in the history

  17. Raise an error if nan_as_null=True

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    4643f9b View commit details
    Browse the repository at this point in the history

  18. Linter corrections

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    a93a46e View commit details
    Browse the repository at this point in the history

  19. Add better test coverage for test_mixed_dtypes and test_dtypes

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    0b231ea View commit details
    Browse the repository at this point in the history

  20. Add better test coverage for test_pandas_roundtrip and add large_memo… 

    …ry mark to test_large_string
    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    d8ab902 View commit details
    Browse the repository at this point in the history

  21. Add pyarrow roundtrip tests and make additional corrections to the co… 

    …de, skip some of pandas roundtrip tests
    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    21af8fb View commit details
    Browse the repository at this point in the history

  22. Correct large string handling and make smaller corrections in convert… 

    … methods definitions
    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    d6140d4 View commit details
    Browse the repository at this point in the history

  23. Change dict arguments in protocol_df_chunk_to_pyarrow

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    e0d1e63 View commit details
    Browse the repository at this point in the history

  24. Update dataframe.num_chunks() method to use to_batches

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    6067fb3 View commit details
    Browse the repository at this point in the history

  25. Check for sentinel values in the datetime more efficently

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    c6eb5f3 View commit details
    Browse the repository at this point in the history

  26. Make bigger changes to how masks and arrays are constructed

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    1a67177 View commit details
    Browse the repository at this point in the history

  27. Import from pandas.api.interchange

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    51dcc49 View commit details
    Browse the repository at this point in the history

  28. Add a check for use_nan, correct test using np.nan and put back check… 

    … for sentinel values deleted by accident
    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    4879ef2 View commit details
    Browse the repository at this point in the history

  29. Add test coverage for pandas -> pyarrow conversion

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    1cbd594 View commit details
    Browse the repository at this point in the history

  30. Rename test_extra.py to test_conversion.py

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    a6b6e54 View commit details
    Browse the repository at this point in the history

  31. Skip pandas -> pyarrow tests for older versions of pandas

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    2e36185 View commit details
    Browse the repository at this point in the history

  32. Add test coverage for sliced table in pyarrow roundtrip

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    4ca948d View commit details
    Browse the repository at this point in the history

  33. Correct the handling of bitpacked booleans

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    719ab88 View commit details
    Browse the repository at this point in the history

  34. Small change in slicing parametrization

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    91ea335 View commit details
    Browse the repository at this point in the history

  35. Add a RuntimeError for boolean and categorical columns in from_datafr… 

    …ame.py if allow_copy=False
    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    c74eb45 View commit details
    Browse the repository at this point in the history

  36. Optimize datetime handling in from_dataframe

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    c137337 View commit details
    Browse the repository at this point in the history

  37. Optimize buffers_to_array in from_dataframe.py

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    1e9cef9 View commit details
    Browse the repository at this point in the history

  38. Apply suggestions from code review - Joris 

    Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
    @AlenkaF @jorisvandenbossche
    AlenkaF and jorisvandenbossche committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    0c539a0 View commit details
    Browse the repository at this point in the history

  39. Add string column back to test_pandas_roundtrip for pandas versions 2… 

    ….0 and higher
    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    6399be3 View commit details
    Browse the repository at this point in the history

  40. Fix linter error

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    9f68fe7 View commit details
    Browse the repository at this point in the history

  41. Remove pandas specific comment for nan_as_null in dataframe.py

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    b926066 View commit details
    Browse the repository at this point in the history

  42. Fix typo boolen -> categorical in categorical_column_to_dictionary

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    5c5d25e View commit details
    Browse the repository at this point in the history

  43. Add a comment for float16 NotImplementedError in validity_buffer_nan_… 

    …sentinel
    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    f2a65a6 View commit details
    Browse the repository at this point in the history

  44. Update validity_buffer_nan_sentinel in python/pyarrow/interchange/fro… 

    …m_dataframe.py

Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>
    @AlenkaF @jorisvandenbossche
    AlenkaF and jorisvandenbossche committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    075e888 View commit details
    Browse the repository at this point in the history

  45. Make change to the offset buffers part of buffers_to_array

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    efa12d6 View commit details
    Browse the repository at this point in the history

  46. Linter correction

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    858cadb View commit details
    Browse the repository at this point in the history

  47. Update the handling of allow_copy keyword

    @AlenkaF
    AlenkaF committed Jan 11, 2023
    Configuration menu
    Copy the full SHA
    e937b4c View commit details
    Browse the repository at this point in the history

Commits on Jan 12, 2023

  1. Fix failing nightly test

    @AlenkaF
    AlenkaF committed Jan 12, 2023
    Configuration menu
    Copy the full SHA
    1b5f248 View commit details
    Browse the repository at this point in the history

  2. Fix the fix for the failing test

    @AlenkaF
    AlenkaF committed Jan 12, 2023
    Configuration menu
    Copy the full SHA
    9139444 View commit details
    Browse the repository at this point in the history