Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[xy] Fix spark dataframe read error. #4856

Merged
merged 1 commit into from
Mar 29, 2024

Conversation

wangxiaoyou1993
Copy link
Member

@wangxiaoyou1993 wangxiaoyou1993 commented Mar 29, 2024

Description

Fix spark dataframe read error.
Do not write json file in spark data directory to avoid read error

java.lang.RuntimeException: file:/home/src/mage_data/.../output_0/type.json is not a Parquet file. 
Expected magic number at tail, but found [109, 101, 34, 125]

How Has This Been Tested?

  • Tested with local spark docker image

Checklist

  • The PR is tagged with proper labels (bug, enhancement, feature, documentation)
  • I have performed a self-review of my own code
  • I have added unit tests that prove my fix is effective or that my feature works
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • If new documentation has been added, relative paths have been added to the appropriate section of docs/mint.json

cc:

@wangxiaoyou1993 wangxiaoyou1993 added the bug Something isn't working label Mar 29, 2024
@wangxiaoyou1993 wangxiaoyou1993 merged commit 703ceab into master Mar 29, 2024
9 checks passed
@wangxiaoyou1993 wangxiaoyou1993 deleted the xiaoyou/fix-spark-dataframe2 branch March 29, 2024 22:19
oonyoontong pushed a commit to bunker-tech/mage-ai that referenced this pull request May 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant