Improve data saving pipeline

[mithi]

Allow updates in the seed function not just inserts

• seed should update data if it already exists in the database
  The current seed function only allows insertion of new data. Currently, if the data exists, it will not update the existing data;seed/populate.js only allows for inserts and not updates.
  The current implementation of db:reset is that the use/db/tables are dropped and recreated and all the data are populated all over again . This is VERY inefficient. https://github.com/mithi/kingdom-rush-graphql/blob/main/src/seed

Refactor python scripts that generate json.

• Refactor python script because they contain a lot of duplicate code.
  https://github.com/mithi/kingdom-rush-graphql/tree/main/scripts

Add new scripts for seeding

• Write a python script to convert generated json to csv

Update pipeline

• Update data saving pipeline

The current pipeline is like this:

1. update yaml
2. python scripts generate json file from yaml
3. Populate the database using the json files with populate.js
4. Populate the csv

It might be so much faster if the pipeline is like this:

1. update yaml
2. python scripts generate json files form yaml
3. use python script to generate csv files from json files
4. populate the database using the csv with the psql command COPY

[mithi]

optimize: Bulk insert into tables

We insert one entry to the database at a time.
To be more efficient, we should instead try to insert in bulk.

EDIT: Converting did not speed up the process significantly. The effect is negligible.

const populateTowers = async ({ dbName = "default", verbose = true } = {}) => {
    const count = await getRepository(Tower, dbName).count()
    console.log("towers in this database:", count)
    if (count !== Number(0)) {
        throw Error(
            `There are already towers in this database. Number of towers: ${count}`
        )
    }

    const towersData: [TowerData] = (<any>towerJson).towers
    const newTowers = towersData.map(towerData => {
        if (verbose) {
            console.log("...")
            console.log(towerData.name, ",", towerData.kingdom)
        }
        return {
            name: towerData.name,
            towerType: mapStringToTowerType[towerData.towerType],
            level: mapIntToLevel[towerData.level],
            kingdom: mapStringToKingdom[towerData.kingdom],
        }
    })

    await getConnection(dbName)
        .createQueryBuilder()
        .insert()
        .into(Tower)
        .values(newTowers)
        .execute()
}