Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(Spreadsheet File Node): Improve CSV parsing #7448

Merged
merged 2 commits into from
Oct 18, 2023
Merged

feat(Spreadsheet File Node): Improve CSV parsing #7448

merged 2 commits into from
Oct 18, 2023

Conversation

netroy
Copy link
Member

@netroy netroy commented Oct 17, 2023 •
edited
Loading

This adds support for

  1. custom delimiters
  2. reading offsets to avoid having to read a large CSV all at once
  3. excluding byte-order-mark

NODE-861
#7443

Workflow Tests

@github-actions
Copy link
Contributor

Great PR! Please pay attention to the following items before merging:

Files matching packages/**:

  • If fixing bug, added test to cover scenario.
  • If addressing forum or Github issue, added link to description.

Files matching packages/**/*.ts:

  • Added unit tests to cover new or updated functionality.

Files matching packages/nodes-base/nodes/**:

  • Added workflow tests for nodes if possible.

Make sure to check off this list before asking for review.

@netroy netroy changed the title feat(Spreadsheet File Node): Improve CSV parsing feat(Spreadsheet File Node): Improve CSV parsing NODE-861 Oct 17, 2023
@netroy netroy changed the title feat(Spreadsheet File Node): Improve CSV parsing NODE-861 feat(Spreadsheet File Node): Improve CSV parsing Oct 17, 2023
@n8n-assistant n8n-assistant bot added n8n team Authored by the n8n team node/improvement New feature or request labels Oct 17, 2023
@codecov
Copy link

codecov bot commented Oct 17, 2023 •
edited
Loading

Codecov Report

Attention: 7 lines in your changes are missing coverage. Please review.

Comparison is base (d8531a5) 33.49% compared to head (7c76ee2) 33.49%.

Additional details and impacted files 
@@           Coverage Diff           @@
##           master    #7448   +/-   ##
=======================================
  Coverage   33.49%   33.49%           
=======================================
  Files        3392     3392           
  Lines      207329   207336    +7     
  Branches    22377    22370    -7     
=======================================
+ Hits        69448    69454    +6     
+ Misses     136757   136754    -3     
- Partials     1124     1128    +4
Files Coverage Δ
...es/nodes-base/nodes/SpreadsheetFile/description.ts 100.00% <ø> (ø)
...nodes/SpreadsheetFile/v2/SpreadsheetFileV2.node.ts 23.14% <22.22%> (+15.24%) ⬆️

... and 4 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Joffcom
Joffcom previously approved these changes Oct 17, 2023
View reviewed changes
@cypress
Copy link

cypress bot commented Oct 17, 2023 •
edited
Loading

Passing run #2509 ↗︎

0 255 0 0 Flakiness 0

Details:

🌳 🖥️ browsers:node18.12.0-chrome107 🤖 netroy 🗃️ e2e/*
Project: n8n Commit: 7c76ee24f2
Status: Passed Duration: 08:49 💡
Started: Oct 18, 2023 2:47 PM Ended: Oct 18, 2023 2:56 PM

Review all test suite changes for PR #7448 ↗︎

@github-actions
Copy link
Contributor

✅ All Cypress E2E specs passed

@netroy
feat(Spreadsheet File Node): Improve CSV parsing
6b7ac0b 
This adds support for
1. custom delimiters
2. reading offsets to avoid having to read a large CSV all at once
3. excluding byte-order-mark
@netroy netroy requested a review from Joffcom October 18, 2023 14:30
@github-actions
Copy link
Contributor

✅ All Cypress E2E specs passed

@netroy netroy merged commit 79f23fb into master Oct 18, 2023
57 of 58 checks passed
@netroy netroy deleted the csv-parsing-improvements branch October 18, 2023 14:57
@github-actions github-actions bot mentioned this pull request Oct 18, 2023
Merged
netroy added a commit that referenced this pull request Oct 18, 2023
@github-actions @netroy
🚀 Release 1.12.0 (#7459)
ef58a23 
# [1.12.0](https://github.com/n8n-io/n8n/compare/n8n@1.11.0...n8n@1.12.0)
(2023-10-18)


### Bug Fixes

* **core:** Add check that queue is defined and remove cyclic dependency
([#7404](#7404))
([45f2ef3](45f2ef3))
* **core:** Do not throw when deleting workflows with executions without
binary-data ([#7411](#7411))
([2b6a15e](2b6a15e))
* **core:** Fix expression with paired item with multi-input node
([#7424](#7424))
([ec14141](ec14141))
* **core:** Fix ignoring crashed executions without event msgs
([#7368](#7368))
([2f4d91b](2f4d91b))
* **core:** Pg-promise de-initialization fix
([#7417](#7417))
([7703904](7703904))
* **core:** Prevent false stalled jobs in queue mode from displaying as
errored ([#7435](#7435))
([e01b9e5](e01b9e5))
* **core:** Prevent undefined issues when restoring binary data
([#7419](#7419))
([46977a2](46977a2))
* **editor:** Fix remote options fetching on every keystroke
([#7320](#7320))
([367255a](367255a))
* **editor:** Open only one tab with plans page
([#7377](#7377))
([c599006](c599006))
* **Google Sheets Node:** Update by row_number, restored 'Handling Extra
Data Option', updated Cell Format default
([#7357](#7357))
([d8531a5](d8531a5))
* **Ldap Node:** Fix issue with connections not closing correctly
([#7432](#7432))
([c3f0be8](c3f0be8))
* **Set Node:** Null should not throw an error
([#7416](#7416))
([e9b6ab0](e9b6ab0))
* **TheHive 5 Node:** Observable encoding in alert > create fix
([#7450](#7450))
([a2d2e3d](a2d2e3d))


### Features

* **core:** Make executions pruning interval configurable
([#7439](#7439))
([40707fa](40707fa))
* **Google Calendar Trigger Node:** Add support for cancelled events
([#7436](#7436))
([9d241a0](9d241a0))
* **HubSpot Trigger Node:** Add support for ticket related events
([#7156](#7156))
([57c6093](57c6093))
* **n8n Form Trigger Node:** New node
([#7130](#7130))
([3ddc176](3ddc176))
* **Spreadsheet File Node:** Improve CSV parsing
([#7448](#7448))
([79f23fb](79f23fb))

Co-authored-by: netroy <netroy@users.noreply.github.com>
@janober
Copy link
Member

janober commented Oct 18, 2023

Got released with n8n@1.12.0

elsmr pushed a commit that referenced this pull request Oct 19, 2023
@netroy @elsmr
feat(Spreadsheet File Node): Improve CSV parsing (#7448)
dc2cf17 
This adds support for
1. custom delimiters
2. reading offsets to avoid having to read a large CSV all at once
3. excluding byte-order-mark

NODE-861
#7443
elsmr pushed a commit that referenced this pull request Oct 19, 2023
@github-actions @netroy @elsmr
🚀 Release 1.12.0 (#7459)
e216ecc 
# [1.12.0](https://github.com/n8n-io/n8n/compare/n8n@1.11.0...n8n@1.12.0)
(2023-10-18)


### Bug Fixes

* **core:** Add check that queue is defined and remove cyclic dependency
([#7404](#7404))
([45f2ef3](45f2ef3))
* **core:** Do not throw when deleting workflows with executions without
binary-data ([#7411](#7411))
([2b6a15e](2b6a15e))
* **core:** Fix expression with paired item with multi-input node
([#7424](#7424))
([ec14141](ec14141))
* **core:** Fix ignoring crashed executions without event msgs
([#7368](#7368))
([2f4d91b](2f4d91b))
* **core:** Pg-promise de-initialization fix
([#7417](#7417))
([7703904](7703904))
* **core:** Prevent false stalled jobs in queue mode from displaying as
errored ([#7435](#7435))
([e01b9e5](e01b9e5))
* **core:** Prevent undefined issues when restoring binary data
([#7419](#7419))
([46977a2](46977a2))
* **editor:** Fix remote options fetching on every keystroke
([#7320](#7320))
([367255a](367255a))
* **editor:** Open only one tab with plans page
([#7377](#7377))
([c599006](c599006))
* **Google Sheets Node:** Update by row_number, restored 'Handling Extra
Data Option', updated Cell Format default
([#7357](#7357))
([d8531a5](d8531a5))
* **Ldap Node:** Fix issue with connections not closing correctly
([#7432](#7432))
([c3f0be8](c3f0be8))
* **Set Node:** Null should not throw an error
([#7416](#7416))
([e9b6ab0](e9b6ab0))
* **TheHive 5 Node:** Observable encoding in alert > create fix
([#7450](#7450))
([a2d2e3d](a2d2e3d))


### Features

* **core:** Make executions pruning interval configurable
([#7439](#7439))
([40707fa](40707fa))
* **Google Calendar Trigger Node:** Add support for cancelled events
([#7436](#7436))
([9d241a0](9d241a0))
* **HubSpot Trigger Node:** Add support for ticket related events
([#7156](#7156))
([57c6093](57c6093))
* **n8n Form Trigger Node:** New node
([#7130](#7130))
([3ddc176](3ddc176))
* **Spreadsheet File Node:** Improve CSV parsing
([#7448](#7448))
([79f23fb](79f23fb))

Co-authored-by: netroy <netroy@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Joffcom Joffcom Joffcom approved these changes

Assignees
No one assigned
Labels
n8n team Authored by the n8n team node/improvement New feature or request Released
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@netroy @janober @Joffcom