Command failed with exit code 10 on Glue Job

[kev-dfs]

Hi everyone

I programmed a processing of data on Jupyter Notebook (SageMaker) with the awswrangler library. This code work perfectly in this enviorement but when I try run it on Glue, the code finish with the next error: Command Failed with exit code 10. This error in the Knowledge Center say that is an error by Memory. Then I runed a memory profile to check how many memory use the process and I find that the process use 25Gb of memory in a "pandas.merge" because the Dataframes are so big (more than 10 Gb each one).
Next, I tryed create "categories" on the some columns for optimize the memory use, but when the code execute the "merge" again, this categories was lose.
¿How can I improve this? Is better change all for a Spark Job (Programmed in Spark)?
I think that someone must haved this problem and could resolved it.

Please I need guidance.
Thanks You.

[jaidisido]

It does seem that you are running into memory issues. I assume you are already using 1 DPU instead of the default 0.0625 DPU in Glue Python shell jobs? If so, then you would probably need to use PySpark instead. The library is not distributed at the moment I am afraid

[sysadmindera]

Please I have the same issue with AWS Glue. Error Message: "Command failed with exit code 10"
Did anyone find a solution for this issue. I use Pandas/Pyspark to process .csv files to Parquet.
Please help.