Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CSV file uses wrong file format (uses JSON instead of CSV) #131

Open
cwegener opened this issue Jan 2, 2024 · 0 comments · May be fixed by #132
Open

CSV file uses wrong file format (uses JSON instead of CSV) #131

cwegener opened this issue Jan 2, 2024 · 0 comments · May be fixed by #132
Labels
bug Something isn't working

Comments

@cwegener
Copy link

cwegener commented Jan 2, 2024

This is a most obvious bug, but somehow I'm the first person to report this?

pipelinewise-target-postgres/target_postgres/db_sync.py

Line 351 in bef5a27

json.dumps(flatten[name], ensure_ascii=False)

The Postgresql COPY command does NOT support reading JSON formatted files. It only supports CSV formatted files (and Binary and Text formats)

https://www.postgresql.org/docs/current/sql-copy.html
@cwegener cwegener added the bug Something isn't working label Jan 2, 2024
cwegener added a commit to cwegener/pipelinewise-target-postgres that referenced this issue Jan 3, 2024
@cwegener
fix: write to file in CSV format, not in JSON
d6ccbc0 
writing in JSON format does not make sense.

The records in Postgres should arrive exactly as they were scraped by a
Tap at the source.

E.g. if a string in the sources has control characters like LF (0x0a),
the same string should also have LF in the postgres target and not '\n'
like JSON format uses.

Fixes transferwise#131
@cwegener cwegener linked a pull request Jan 3, 2024 that will close this issue
Open
cwegener added a commit to cwegener/pipelinewise-target-postgres that referenced this issue Jan 3, 2024
@cwegener
fix: write to file in CSV format, not in JSON
d00e4b9 
writing in JSON format does not make sense.

The records in Postgres should arrive exactly as they were scraped by a
Tap at the source.

E.g. if a string in the sources has control characters like LF (0x0a),
the same string should also have LF in the postgres target and not '\n'
like JSON format uses.

Fixes transferwise#131
cwegener added a commit to cwegener/pipelinewise-target-postgres that referenced this issue Jan 3, 2024
@cwegener
fix: write to file in CSV format, not in JSON
02b70ea 
writing in JSON format does not make sense.

The records in Postgres should arrive exactly as they were scraped by a
Tap at the source.

E.g. if a string in the sources has control characters like LF (0x0a),
the same string should also have LF in the postgres target and not '\n'
like JSON format uses.

Fixes transferwise#131
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: write to file in CSV format, not in JSON cwegener/pipelinewise-target-postgres
1 participant
@cwegener