Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace csv-parser with papaparse #19739

Merged
merged 17 commits into from
Oct 12, 2023

Conversation

u12206050
Copy link
Contributor

@u12206050 u12206050 commented Sep 19, 2023

Replaces csv-parser with papaparse.

This allows for supporting better detection of separators such as ;
Value parsing also seems to be easier with papaparse.

Fixes #15874

@changeset-bot
Copy link

changeset-bot bot commented Sep 19, 2023

🦋 Changeset detected

Latest commit: ff7c6fc

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 2 packages
Name Type
@directus/api Minor
directus Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@u12206050 u12206050 marked this pull request as ready for review September 19, 2023 18:46
Copy link
Contributor

@DanielBiegler DanielBiegler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks so much for the PR! ❤️

api/package.json Outdated Show resolved Hide resolved
api/package.json Outdated Show resolved Hide resolved
DanielBiegler
DanielBiegler previously approved these changes Sep 20, 2023
Copy link
Contributor

@DanielBiegler DanielBiegler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yay, semicolons work now! Tested the following basic example:

"bool";"msg";"name"
true;"enabled first yea";"first"
null;"null second hello";"second"
false;"disabled third yeasssss";"third"

But skipped values are now interpreted as an empty string: "" which fails for other datatypes. In my boolean example from above if you skip the null (which is how our export does it) the import fails.

On v10.6.2 without papaparser left out values work, as they're null.

So the following works on v10.6.2 but not with this PR:

"bool","msg","name"
true,"enabled first yea","first"
,"null second hello","second"
false,"disabled third yeasssss","third"

Here is the error message you'll see:

[INTERNAL_SERVER_ERROR] insert into "titles" ("bool", "msg", "name") values ($1, $2, $3) returning "id" - invalid input syntax for type boolean: ""

@paescuj paescuj marked this pull request as draft September 21, 2023 10:27
@u12206050 u12206050 marked this pull request as ready for review September 21, 2023 13:45
@paescuj paescuj marked this pull request as draft September 21, 2023 13:59
@u12206050 u12206050 marked this pull request as ready for review September 23, 2023 18:58
@jaads jaads removed their request for review September 28, 2023 07:20
@paescuj
Copy link
Member

paescuj commented Oct 12, 2023

With the latest changes it behaves the same as before with csv-parser, meaning it's backwards-compatible and from my point of view nothing speaks against switching to papaparse anymore.

@DanielBiegler Can you check out again?

@paescuj paescuj removed the request for review from azrikahar October 12, 2023 15:27
Copy link
Contributor

@DanielBiegler DanielBiegler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did multiple tests with multiple different configs and formats and relations and everything worked! ❤️ LGTM 🚀 🚀 🚀

@paescuj paescuj enabled auto-merge (squash) October 12, 2023 17:07
@paescuj paescuj merged commit bf05c8f into directus:main Oct 12, 2023
7 checks passed
@github-actions github-actions bot added this to the Next Patch Release milestone Oct 12, 2023
br-rafaelbarros pushed a commit to personal-forks/directus-source that referenced this pull request Nov 7, 2023
Co-authored-by: Daniel Biegler <DanielBiegler@users.noreply.github.com>
Co-authored-by: Pascal Jufer <pascal-jufer@bluewin.ch>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 1, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Format of CSV Import (Configuration of Delimiter / Separator)
4 participants