Incorporate Fixed Vocabularies

[Iain-S]

We may want to copy some tables wholesale. Use cases might include something like the omop concept table in its entirety.

[Iain-S]

Perhaps a different SQLSynthGen base class could take care of this

class conceptGenerator(VocabGenerator):
   pass

sorted_generators = [
   ...,
   conceptGenerator,
  ...,
]

We would also have to change the logic of the populate function.

[myyong]

When I think of fixing the vocabulary, I think of using the hydef values from Jon: https://github.com/alan-turing-institute/TSDiP/issues/21

[Iain-S]

Markus and I agree that it would be nice if no commands require you to have access to both databases at once. For that reason, we may want to export vocab tables either

• during the make-generators command or
• in a separate make-vocab command

Then, for the create-data step (or create-vocab, perhaps) we can have a generator that simply reads in a whole file. Things to be aware of are the file format with which to store the vocab. Maybe csv files or maybe pgdumps with insert statements.

[Iain-S]

If we can presume that vocabulary tables don't ever depend on private data, we can add functionality to make-gens so that it

1. exports the raw vocab data
2. makes file-upload generators for the vocab tables
3. creates generators for the non-vocab tables

and we can split the create-data step into two steps

1. create-vocab which runs the file-upload generators
2. create-data which runs the other generators

so that create-tables and create-vocab should only need to be called once.

To begin with, we can specify vocab tables on the CLI and then merge with Markus' config file

sqlsynthgen make-gens --vocab=table1,table2 orm.py > ssg.py

At this point, we are potentially creating more than one output file so we may want to change how the command works (e.g. writing to pre-specified file names and erroring if any of those files already exist).

[mhauru]

Closed by #41