Skip to content

Node.js script that concatenates unlimited CSV's into a single file using Streams.

Notifications You must be signed in to change notification settings

kastorcode/large-csv-stream-concat-nodejs

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

1 Commit
 
 
 
 
 
 
 
 

Repository files navigation

Large CSV Concat with Node.js Streams

👷 Developed by Matheus Ramalho de Oliveira
🔨 Brazilian Software Engineer
🏡 Goiânia, Goiás, Brasil
✉️ kastorcode@gmail.com
👍 instagram.com/kastorcode


This script uses the best possible practice for working with files on demand in Node.js. We will concatenate unlimited gigabytes of information from different files converting from CSV to JSON, taking the data we are interested in and converting it back from JSON to CSV, all using streams.


Installation and execution

  1. Make a clone of this repository;
  2. Create a folder called dataset;
  3. Download 2 data packs: https://kaggle.com/datasets/stackoverflow/so-survey-2017 and https://kaggle.com/datasets/stackoverflow/stack-overflow-2018-developer-survey;
  4. Extract the data into the dataset folder;
  5. Open the project folder in a terminal;
  6. Run yarn to install dependencies;
  7. Run yarn start to start the concatenation.

<kastor.code/>