-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add support for writing additional file types beyond parquet #101
Comments
Can you point to where I should start looking to explore this issue? |
Definitely! All the file io is in this file: https://github.com/launchflow/buildflow/blob/main/buildflow/runtime/ray_io/file_io.py The enum here has the list of file types we support: https://github.com/launchflow/buildflow/blob/main/buildflow/runtime/ray_io/file_io.py#L18 And in the You should be able to mimic the tests here with any additional file types also: https://github.com/launchflow/buildflow/blob/main/buildflow/runtime/ray_io/file_io_test.py If you have any other questions let me know! |
Thanks, I an trying to test my changes and found that the existing tests are using |
We mostly just did that from preference. Previous it was nice to use to setup ray specifics in the setUpClass methods, but we moved that to a pytest fixture, but it should play well with pytest I believe. My typical workflow for a fresh install is (running from the root directory):
|
Thanks, I got the tests working. I have send a pull request #137 . But the checks are not running. And I was not able to add reviewer in there. |
Hmm interesting probably an issue with our permission set up. Let me try and update those |
Alright turns out it was an issue with when the workflow runs. I updated it in #138 to run on PRs. So I think if you pull down the latest changes it should work. (also added a CODEOWNERS to auto assign reviews) |
With #142 , filesink should now supports CSV and JSON |
Some more file types: JSON, Avro, CSV
The text was updated successfully, but these errors were encountered: