pyarrow 3.0.0 - glue wheel, lambda layer, and pip install awswrangler

[jarretg]

With awswrangler 2.4.0, I see that pyarrow 3.0.0 is supported. When I do a "pip install awswrangler", it remains at 2.0.0 in my notebook. Shouldn't that upgrade pyarrow to 3.0.0 also? How do I get the pyarrow to upgrade to 3.0.0? I found a data issue that seems to be resolved in pyarrow 3.0.0, but when installing awswrangler, the pyarrow version stays at 2.0.0. I need pyarrow 3.0.0 when using the Lambda layer and Glue wheel as well.

Thanks in advance!

[igorborgest]

Hi @jarretg!

what is your Python version?

[igorborgest]

Try to uninstall any existing pyarrow version before install wrangler.

[jarretg]

Hi @igorborgest, hope all is well!

My Sagemaker notebook is showing python version 3.6.11. For the lambda layer, I choose the 3.7 runtime. Glue, I have the "Python 3, Glue Version 1.0".

[j0bekt01]

Try to uninstall any existing pyarrow version before install wrangler.
wouldn't it be easier just to update the requirements.txt file pyarrow>=3.0.0?

[jarretg]

I can try the uninstall tomorrow.

How do I get Glue and Lambda to use pyarrow 3.0.0 as well?

[igorborgest]

@jarretg

For Glue and Lambda is comes naturally, you just need to use the latest Glue wheel or Lambda layer provided in the latest release.

SageMaker already has pyarrow==2.0.0 installed by default and pip install awswrangler will not update it automatically because the 2.0.0 is also accepted. So you only need to install/upgrade pyarrow by yourself with pip install -U pyarrow or pip install pyarrow==3.

[igorborgest]

@j0bekt01

wouldn't it be easier just to update the requirements.txt file pyarrow>=3.0.0?

We don't want to force everyone to upgrade pyarrow, and there is some platforms that don't support the version 3 yet (e.g. #546 #547 #549 ).

[j0bekt01]

@j0bekt01

wouldn't it be easier just to update the requirements.txt file pyarrow>=3.0.0?

We don't want to force everyone to upgrade pyarrow, and there is some platforms that don't support the version 3 yet (e.g. #546 #547 #549 ).

makes sense, thanks!

[raisingknights]

Below is the boot script which I using in EMR, I am also getting the same error. could you please let me know what will help to resolve this

I tried adding pyarrow==2.0.0 awswrangler in install script still is failling

set -x -e

echo -e 'export PYSPARK_PYTHON=/usr/bin/python3
export HADOOP_CONF_DIR=/etc/hadoop/conf
export SPARK_JARS_DIR=/usr/lib/spark/jars
export SPARK_HOME=/usr/lib/spark' >> $HOME/.bashrc && source $HOME/.bashrc

sudo python3 -m pip install awscli boto3 spark-nlp wordcloud image

set +x
exit 0

[kukushking]

@raisingknights can you clarify exactly what is failing, and what is the error message?