Welcome to my school project for the class dataproccesing.
This project is made for a school assignment where I needed to process data from a database to a consumer application.
The goal is to make a REST API that uses XML and JSON.
We need to make an application that uses the REST API, the apps need to switch between XML and JSON.
- Introduction
- Table of Contents
- Tools used
- Database Import
- REST API
- Consumer application
- How to install the database
| Tool | Used for | Website |
|---|---|---|
| XAMPP | Hosting local MySQL DB | Website |
| HeidiSQL | Importing SQL dump | Website |
| EmEditor | Editing and splitting .sql file | Website |
| SSMA | Migration tool from MySQL to MSSQL | Website |
| SSMS | Management Studio for MSSQL | Website |
| Swagger | REST API Documentation | Website |
I downloaded the Dataset here: https://steam.internet.byu.edu/
After downloading the dataset I found out that importing a 160 GB dataset is a challenge in itself, not even talking about making a MySQL Dump into a working MS SQL Database.
I used the following command to import it into PhpMyAdmin
mysql --init-command="SET SESSION FOREIGN_KEY_CHECKS=0;SET UNIQUE_CHECKS=0;" -u root -p DataBaseNamePlaceholder < C:\NameOfTheSqlFile
But quickly switched to HeidiSQL mainly to check the progress. HeidiSQL also splits the file into multiple queries, see the GIF for an example.
The SQL file (160 GB) is too big, even tho HeidiSQL splits the file into multiple queries.
I solved this issue by using EmEditor.
EmEditor is made to load big files and also works great to edit big files.
But the feature I used the most is the splitting tool, with this I was able to split the SQL file down into 8 files.
Each file consists out of 1 table, except for some small tables.
With the smaller SQL files I was able to import the tables I need into MySQL with the help of HeidiSQL.
The SQL file is imported in MySQL, but the end goal is to have it in MSSQL.
I used SSMA (SQL Server Migration Assistant) to migrate the MySQL datbase into MSSQL
I used ASP.NET Core Web API for the API. It was the first time using Core instead of Net and the first time I made a REST API.
I used Swagger to create documentation for my REST API which saved me a lot of time which is really great.
- First step was making models so I could scaffold them to create controllers.
- Second step was to add pagination so the API wouldn't crash with the huge amount of rows in the database.
- As of last I created one GET, POST, PUT and DELETE for each database table as are the requirements from school.
I didn't have enough time to create the application. Lucky enough I got a 7.5/10 so I won't be making the application.
