Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[master < T0849] Add HTTP+GZIP support to LOAD CSV #1027

Merged
merged 11 commits into from
Jun 26, 2023

Conversation

Ignition
Copy link
Contributor

@Ignition Ignition commented Jun 20, 2023

Added network and compression support

Network: http, https, tftp, ftp, and telnet
Compression: gzip, and bzip2

[master < Task] PR

  • Check, and update documentation if necessary
  • Provide the full content or a guide for the final git message

To keep docs changelog up to date, one more thing to do:

  • Write a release note here, including added/changed clauses -> "Added network and compression support to LOAD CSV"
  • Tag someone from docs team in the comments

closes #849

@Ignition Ignition force-pushed the MG_849_add_http_gzip_support_to_load_csv branch 3 times, most recently from 44103f1 to c1d819f Compare June 21, 2023 09:23
libs/setup.sh Show resolved Hide resolved
src/requests/requests.cpp Outdated Show resolved Hide resolved
src/utils/csv_parsing.cpp Outdated Show resolved Hide resolved
@Ignition Ignition force-pushed the MG_849_add_http_gzip_support_to_load_csv branch from 16d8431 to 3aeb3f5 Compare June 22, 2023 10:05
@gitbuda gitbuda added the feature feature label Jun 22, 2023
@gitbuda gitbuda added this to the mg-v2.9.0 milestone Jun 22, 2023
@Ignition Ignition force-pushed the MG_849_add_http_gzip_support_to_load_csv branch 2 times, most recently from b4df0a4 to 850f8f2 Compare June 22, 2023 13:54
@Ignition Ignition marked this pull request as ready for review June 22, 2023 16:34
@Ignition Ignition requested a review from gitbuda June 22, 2023 16:34
@Ignition
Copy link
Contributor Author

@vpavicic this will need documentation changes.

Copy link
Contributor

@Josipmrden Josipmrden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small number of comments

src/csv/fuzz/fuzz_reader.cpp Show resolved Hide resolved
src/csv/fuzz/fuzz_reader.cpp Outdated Show resolved Hide resolved
src/csv/include/csv/parsing.hpp Show resolved Hide resolved
@Ignition Ignition force-pushed the MG_849_add_http_gzip_support_to_load_csv branch from 850f8f2 to 50e63d7 Compare June 23, 2023 16:19
@Ignition Ignition requested a review from Josipmrden June 24, 2023 12:27
@vpavicic
Copy link
Contributor

vpavicic commented Jun 26, 2023

@vpavicic this will need documentation changes.

We need to change the following pages:

https://memgraph.com/docs/memgraph/import-data/load-csv-clause
https://memgraph.com/docs/cypher-manual/clauses/load-csv

@Ignition Would you mind creating a PR in the docs repo or hopping on a call today to explain the feature in detail?

The changelog message would be something along the lines of:

Now you can import data from web-hosted CSV files by passing the URL in the LOAD CSV clause. You can also import files compressed with GZIP or BZIP2 algorithms.

@Ignition Ignition force-pushed the MG_849_add_http_gzip_support_to_load_csv branch from f2fcd73 to 0ce768d Compare June 26, 2023 09:27
@Ignition Ignition requested a review from antejavor June 26, 2023 09:37
Copy link
Member

@gitbuda gitbuda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me, I'll add an issue/task to implement e2e test for this 😄

@gitbuda gitbuda merged commit 3b781bf into master Jun 26, 2023
6 checks passed
@gitbuda gitbuda deleted the MG_849_add_http_gzip_support_to_load_csv branch June 26, 2023 17:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement HTTP + GZIP support as part of LOAD CSV to avoid copying
5 participants