Scaling up
No due date
50% complete
Geneve works similarly to a language compiler. Briefly:
- The data specs are parsed and translated into abstract syntax trees (AST)
- The ASTs are traversed and the fields constraints are collected in an intermediate representation (IR)
- The constraints of each field are solved, if possible, and stored in sets of field-value pairs
- The sets of field-value pair…
Geneve works similarly to a language compiler. Briefly:
- The data specs are parsed and translated into abstract syntax trees (AST)
- The ASTs are traversed and the fields constraints are collected in an intermediate representation (IR)
- The constraints of each field are solved, if possible, and stored in sets of field-value pairs
- The sets of field-value pairs are translated to one or more documents
- The generated documents are ingested into the Stack
Whereas steps 1-2 are executed once, steps 3-5 are iterative and present a ceiling that is hit depending on:
- number and complexity of the relations that generated data needs to satisfy (step 1 and 2)
- number and complexity of the constraints problems to be solved for generating the documents (step 3)
- number of documents to be generated (iteration of steps 3-5)
This milestone seeks solutions to the scalability limits due to having all the five steps above implemented in Python.