Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

`SAVE AND USE' should work for CSV. #332

Open
goodwanghan opened this issue Jun 20, 2022 · 0 comments
Open

`SAVE AND USE' should work for CSV. #332

goodwanghan opened this issue Jun 20, 2022 · 0 comments
Labels
bug Something isn't working

Comments

@goodwanghan
Copy link
Collaborator

Originally posted by @keiranmraine in #331 (comment)

File c:\Users\XXX\.venv\lib\site-packages\fugue\workflow\workflow.py:1518, in FugueWorkflow.run(self, *args, **kwargs)
   1516         if ctb is None:  # pragma: no cover
   1517             raise
-> 1518         raise ex.with_traceback(ctb)
   1519     self._computed = True
   1520 return DataFrames(
   1521     {
...
    231         pdf = _safe_load_csv(
    232             p.uri, **{"index_col": False, "header": None, "names": columns, **kw}
    233         )

ValueError: columns must be set if without header

This is because SAVE AND USE on csv files does not take the parameters sep and header.

Solution:

  1. It is recommended not to use CSV when you want to use SAVE AND USE because CSV has so many special parameters, they are not consistent cross frameworks, and they are not consistent between read and write, it is very hard to unify them, sometimes impossible.
  2. Just try to use the parameters when loading, if it doesn't work, we should raise an exception, and also warn people that it's not a good idea to save and use csv.

We will implement 2, but please keep 1 in mind.

@goodwanghan goodwanghan added the bug Something isn't working label Jun 20, 2022
@goodwanghan goodwanghan added this to the 0.7.0 milestone Jun 20, 2022
@goodwanghan goodwanghan removed this from the 0.7.0 milestone Jul 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant