Skip to content

upgrading pyarrow to fix the numpy 1.21.0 broken changes and fixing integ tests#111

Closed
NikhilRaverkar wants to merge 2 commits intomasterfrom
master_pyarrow_fix
Closed

upgrading pyarrow to fix the numpy 1.21.0 broken changes and fixing integ tests#111
NikhilRaverkar wants to merge 2 commits intomasterfrom
master_pyarrow_fix

Conversation

@NikhilRaverkar
Copy link
Copy Markdown

@NikhilRaverkar NikhilRaverkar commented May 2, 2022

Issue #, if available:

  1. Updated pyarrow version to 3.0.0 to fix the broken changes with numoy 1.21.0
  2. Fixed broken integ tests

Description of changes:
Fixed pyarrow version

Testing:
Verified that error does not exist anymore after the upgrade:

>>> import pandas as pd
>>> df = pd.DataFrame({'x': [1,2]})
>>> df.to_parquet('test.parquet', engine='pyarrow')
>>> print(pd.__version__)
1.1.3
>>> print(pyarrow.__version__)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'pyarrow' is not defined
>>> import pyarrow
>>> print(pyarrow.__version__)
4.0.0
>>> import numpy
>>> print(numpy.__version__)
1.21.0
>>> 

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

' --mount type=volume,source=dynamic_endpoint_module_volume,target=/user_module,readonly'
# ' --mount type=volume,source=dynamic_endpoint_model_volume,target=/opt/ml/model,readonly'
# ' --mount type=volume,source=dynamic_endpoint_module_volume,target=/user_module,readonly'
' -v {}:/opt/ml/model'
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for my understanding, why were these arguments module_dir and model_dir added with these changes?

Copy link
Copy Markdown
Author

@NikhilRaverkar NikhilRaverkar May 3, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the confusion, these changes are unrelated to the pyarrow upgrade. The integ tests are currently broken so this change is fixing the integ tests as well. Updated description and title

@@ -55,19 +55,24 @@ def modulevolume():

@pytest.fixture(scope='module', autouse=True)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we also want to update the version in test setup(tox.ini)?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! updated

@NikhilRaverkar NikhilRaverkar requested a review from haixiw May 3, 2022 18:34
@NikhilRaverkar NikhilRaverkar changed the title upgrading pyarrow to fix the numpy 1.21.0 broken changes upgrading pyarrow to fix the numpy 1.21.0 broken changes and fixing integ tests May 3, 2022
@NikhilRaverkar NikhilRaverkar removed the request for review from haixiw May 3, 2022 18:37
@NikhilRaverkar
Copy link
Copy Markdown
Author

Closing this since PR#110 is upgrading the version. Will create a new PR for the new version

@NikhilRaverkar NikhilRaverkar deleted the master_pyarrow_fix branch May 10, 2022 18:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants