Welcome to the Quorum Mid Level Full Stack Developer Technical Challenge repository

Summary:

• What was developed
• Technologies used
• Execute the project
• Documentation
• Next Steps
• Final considerations

---

What was developed

An API for processing CSV files using Node.js, JavaScript and Express. Also develop a front end using HTML and CSS to display the results.

Back to the top ☝

---

Technologies used

Back to the top ☝

---

Execute the project

Clone the repository

git clone https://github.com/Kecbm/quorum.git

Install dependencies

cd quorum
npm install

Execute the API for generating CSV files

cd quorum
npm start

At this moment, the files bills-support-oppose-count.csv and legislators-support-oppose-count.csv are already generated in the results folder. To generate them again it is necessary to delete the files in the results folder and run the project again, with the npm start command.

Run the tests

cd quorum
npm test

Access the frontend

Execute the Server

cd quorum
npm run serve

The server will start and listen on port 3000. You can access the frontend in http://localhost:3000.

Back to the top ☝

---

Documentation

The documentation is intended to provide a clear and detailed overview of the developed API, facilitating understanding and future maintenance.

CSV File Processing API Documentation

Description

This module processes CSV files related to votes, vote results, bills, and legislators. It generates two output CSV files with statistics on support and opposition for bills and support from legislators. This documentation section is about the index.js file.

Functionalities

• Reading CSV Files: The module reads CSV files containing information about votes, vote results, bills and legislators;
• Data Processing: Updates and organizes data on votes and legislators;
• Report Generation: Creates CSV files with detailed statistics on bill support and legislator support.

Code Architecture

Imports

const fs = require('fs');
const csv = require('csv-parser');
const createCsvWriter = require('csv-writer').createObjectCsvWriter;

• fs: Node.js module for manipulating files and directories;
• csv-parser: Library for parsing CSV files;
• csv-writer: Library for creating and writing CSV files.

CSV Writers Configuration

const legislatorSupportWriter = createCsvWriter({
    path: 'results/legislators-support-oppose-count.csv',
    header: [
        { id: 'id', title: 'id' },
        { id: 'name', title: 'name' },
        { id: 'num_supported_bills', title: 'num_supported_bills' },
        { id: 'num_opposed_bills', title: 'num_opposed_bills' }
    ]
});

const billSupportWriter = createCsvWriter({
    path: 'results/bills-support-oppose-count.csv',
    header: [
        { id: 'bill_id', title: 'id' },
        { id: 'title', title: 'title' },
        { id: 'supporter_count', title: 'supporter_count' },
        { id: 'opposer_count', title: 'opposer_count' },
        { id: 'sponsor', title: 'primary_sponsor' }
    ]
});

• legislatorSupportWriter: Configures the writer to generate the CSV file with information about legislators' support and opposition;
• billSupportWriter: Configures the writer to generate CSV file with information about support and opposition to bills.

processCSV function

const processCSV = (filePath, onData) => new Promise((resolve, reject) => {
    fs.createReadStream(filePath)
        .pipe(csv())
        .on('data', onData)
        .on('end', resolve)
        .on('error', reject);
});

• filePath: Path of the CSV file to be read;
• onData: Callback function that processes each line of data;
• resolve: Resolves the promise when processing is complete;
• reject: Rejects the promise in case of error.

Data Update Functions

const updateVoteData = (votes, data) => {
    const voteData = votes.get(data.vote_id) || { supporter_count: 0, opposer_count: 0 };
    data.vote_type === '1' ? voteData.supporter_count++ : voteData.opposer_count++;
    votes.set(data.vote_id, voteData);
};

const updateLegislatorData = (legislatoresVotes, data) => {
    let legislatorVotes = legislatoresVotes.get(data.legislator_id) || { num_supported_bills: 0, num_opposed_bills: 0 };
    data.vote_type === '1' ? legislatorVotes.num_supported_bills++ : legislatorVotes.num_opposed_bills++;
    legislatoresVotes.set(data.legislator_id, legislatorVotes);
};

• updateVoteData(votes, data): Updates the number of supporters and opponents of a bill based on the vote results;
• updateLegislatorData(legislatoresVotes, data): Updates the number of bills supported and opposed by a legislator based on the vote result.

Main Function processVotes

const processVotes = async () => {
    const votes = new Map();
    const storeBillVotes = {};
    const storeSponsorVote = {};
    const legislatoresVotes = new Map();
    const legislatorSupport = [];

    await processCSV('data/votes.csv', data => {
        votes.set(data.id, { bill_id: data.bill_id, supporter_count: 0, opposer_count: 0 });
        storeBillVotes[data.bill_id] = data.id;
    });

    await processCSV('data/vote_results.csv', data => {
        updateVoteData(votes, data);
        updateLegislatorData(legislatoresVotes, data);
    });

    await processCSV('data/bills.csv', data => {
        const vote_id = storeBillVotes[data.id];
        const voteData = votes.get(vote_id) || {};
        votes.set(vote_id, { ...voteData, title: data.title, sponsor_id: data.sponsor_id, sponsor: "Unknown" });
        storeSponsorVote[data.sponsor_id] = vote_id;
    });

    await processCSV('data/legislators.csv', data => {
        const vote_id = storeSponsorVote[data.id];
        if (vote_id) {
            const voteData = votes.get(vote_id);
            votes.set(vote_id, { ...voteData, sponsor: data.name });
        }

        const legislatorData = legislatoresVotes.get(data.id) || { num_supported_bills: 0, num_opposed_bills: 0 };
        legislatorSupport.push({
            id: data.id,
            name: data.name,
            num_supported_bills: legislatorData.num_supported_bills,
            num_opposed_bills: legislatorData.num_opposed_bills
        });
    });

    await Promise.all([
        legislatorSupportWriter.writeRecords(legislatorSupport),
        billSupportWriter.writeRecords([...votes.values()])
    ]);

    console.log('CSV files processed successfully.');

    processVotes().catch(error => console.error('Error processing CSV files:', error.message));
};

• processVotes(): Main function that orchestrates the processing of CSV files, updates the data and generates the output CSV files;
• processCSV(filePath, onData): Used to read and process CSV files;
• await Promise.all([...]): Ensures that CSV files are written before ending the process.

CSV File Structure

data/votes.csv

• Fields: id, bill_id, vote_type;
• Description: Data on individual votes, indicating the bill and the type of vote (support or opposition).

data/vote_results.csv

• Fields: vote_id, legislator_id, vote_type;
• Description: Vote results, associating individual votes with legislators and the type of vote.

data/bills.csv

• Fields: id, title, sponsor_id;
• Description: Information about bills, including title and sponsor identifier.

data/legislators.csv

• Fields: id, name;
• Description: Information about legislators, including identifier and name.

Log Messages

• "CSV files processed successfully.": Indicates that the processing and writing of CSV files were completed successfully;
• "Error processing CSV files:": Error message when a failure occurs in processing CSV files.

Comments

• Maintenance: The code is structured to be easy to maintain and modify, with reusable functions and a modular approach;
• Performance: Reading and writing CSV files is done asynchronously to improve performance and responsiveness.

Server Documentation

Overview

This Express server manages CSV file processing tasks. It includes functionality to start the processing of CSV files and list available CSV files in the results directory. This documentation section is about the server.js file.

Dependencies

• express: Web framework for Node.js;
• child_process: Node.js module to execute shell commands;
• fs: Node.js module for file system operations.
• path: Node.js module for handling file paths.

Endpoints

POST /start

• Description: Deletes specific CSV files from the results directory and starts the processing of CSV files by executing an npm start command;
• Response:
• 200 OK: If the processing starts successfully;
• 500 Internal Server Error: If there is an error starting the processing.

Example Request:

    POST /start

Example Response:

"Processing of CSV files started successfully."

Note: The following CSV files are deleted before starting the processing:

• legislators-support-oppose-count.csv
• bills-support-oppose-count.csv

GET /csv-files

• Description: Lists all CSV files present in the results directory and returns their content;
• Response:
• 200 OK: Returns a JSON array with the filenames and contents of the CSV files;
• 500 Internal Server Error: If there is an error listing the files.

Example Request:

    GET /csv-files

Example Response:

[
  {
    "filename": "legislators-support-oppose-count.csv",
    "content": "id,name,num_supported_bills,num_opposed_bills\n1,John Doe,5,2\n2,Jane Smith,3,4"
  },
  {
    "filename": "bills-support-oppose-count.csv",
    "content": "id,title,supporter_count,opposer_count,primary_sponsor\n1,Bill A,10,2,John Doe\n2,Bill B,5,7,Jane Smith"
  }
]

Back to the top ☝

---

Next Steps

Aiming to offer continuous improvement Software, considering the MVP development where the analysis of customer feedback is carried out with each new delivery and adjustments to existing functionalities or development of new functionalities, below I list a series of tasks to be carried out in the project in the future, which forms an initial product backlog. Feel free to contribute to this list.

• Build a frontend in React;
• Refactor the API to use the MSC architecture;
• Replace CSS with Tailwind;
• Develop new application routes;
• Make the application responsive;
• Develop the dark theme;
• Develop the Not Found page;
• Deploy the application;
• Build the database for the API;
• Dockerize the frontend, backend and database;
• Develop frontend and server tests;
• Develop integration tests;
• Consume data dynamically;
• Make the analysis CSV available for download;

Back to the top ☝

---

Final considerations

Answers

1

Time complexity depends on the number of lines in the CSV files. If we have to read and process multiple files, the time it takes to do so increases with the number of lines in the largest file. We can say that the time needed is proportional to the number of lines in the largest file, which is what most affects speed. So if the largest file has n lines, the time complexity is O(n). This means that the time to process files increases linearly with the size of the largest CSV file;

2

If we need to add a new column, such as 'Co-sponsors', I would adjust the code to look for this new information. For the 'Bill voted on date' column, if it is necessary to count votes before or after a specific date, I would compare the dates. This ensures that the code counts the correct votes according to the date. In short, I would just update the code to handle new columns and do date comparisons correctly;

3

The current solution can be adapted to work with lists of legislators or bills instead of CSV files, maintaining efficiency in data processing. The API was designed to handle large volumes of data using streaming, as demonstrated in the fs.createReadStream snippet on line 27 of the index.js file.

Data Input: Instead of reading data from CSV files, the data would be passed directly as lists in JSON format or another appropriate data structure.

Data Processing: Instead of processCSV, we will use functions that iterate over the provided lists. We will use array methods like forEach or map to process the data.

Data Update: The updateVoteData and updateLegislatorData functions would remain largely the same, being called within the iteration methods to update vote and legislator counts.

Writing of Results: After processing, the results would be written to CSV files using createObjectCsvWriter as before, ensuring the data is formatted correctly for export.

4

I started developing the solution at 9 pm on 07/18/2024 and finished development at 12 pm on the same day. It took me a while to commit the resolution because I was focused on developing the challenge's README.

Back to the top ☝

---

Project developed by Klecianny Melo 😁