Data Management

This section discusses how we manage our data in the different flows.

Student Portal to Scrapper

The Scrapper gets the json file with announcements. From the json file, it creates an outline json file for the structure of the csv. After that, we use our json-to-csv converter to create a csv file with the json file from the Portal.

Scrapper to DB Server

The DB Server receives the announcements in csv format from the Scrapper. Below is a list of the columns in the csv file. We chose certain information from the csv created from the json file and the columns are subject to change in the future if we decide certain information is useful or not.

• category_id
• category_name
• created_at
• email
• fullname
• id
• message
• publish_date
• title
• topic
• updated
• updated_at