refactor faker parsing and enable array relationships

[chuck-alt-delete]

What type of PR is this? (check all applicable)

• ♻️ Refactor
• ✨ Feature
• 🐛 Bug Fix
• 👷 Optimization
• 📝 Documentation Update
• 🚩 Other

Description

This PR simplifies parsing of faker methods, making more methods available, and making them available in a way that is consistent with faker API docs (see tests/array.json).

As a bonus of parsing faker methods this way, we also gain the ability to generate arrays with the faker.helpers.uniqueArray method and specify an array as a parent field in a relationship.

This is a breaking change because now you would have to give a string that represents calling a fully qualified faker method, e.g. "faker.datatype.number()" instead of "datatype.number".

⚠️ This is a work in progress. If this direction is acceptable, the breaking change means I would need to update all the tests and examples.

💡 In this PR thus far, only strings that represent javascript prefixed with faker. can be executed (i.e. faker methods). We may consider widening the scope of what javascript can be passed with strings. If we do so, it would give increased flexibility into what fake data can be generated. However, it would make the application susceptible to a malicious input schema file. This won't be addressed in this PR but is food for thought if we find ourselves needed to expand functionality beyond the faker API.

Related Tickets & Documents

closes #63

Added to documentation?

• 📜 readme
• 🙅 no documentation needed

[chuck-alt-delete]

Here is it in action:

datagen   -dr -n 1 -s tests/array.json -f avro --prefix chuck -d

output:

{
  format: 'avro',
  number: 1,
  dryRun: true,
  schema: 'tests/array.json',
  prefix: 'chuck',
  debug: true
}

✔  Parsing JSON schema...  

[
  {
    _meta: { key: 'id', topic: 'test', relationships: [Array] },
    id: 'faker.datatype.uuid()',
    arr: 'faker.helpers.uniqueArray(faker.datatype.uuid, 5)'
  },
  { _meta: { topic: 'test2', key: 'id' }, id: 'faker.datatype.uuid()' }
]

✔  Avro Schema for topic chuck_test:  
 {
  "name": "chuck_test",
  "type": "record",
  "fields": [
    {
      "name": "id",
      "type": "string"
    },
    {
      "name": "arr",
      "type": {
        "type": "array",
        "items": "string"
      }
    }
  ],
  "namespace": "com.materialize"
}


✔  Avro Schema for topic chuck_test2:  
 {
  "name": "chuck_test2",
  "type": "record",
  "fields": [
    {
      "name": "id",
      "type": "string"
    }
  ],
  "namespace": "com.materialize"
}


✔  Dry run: Skipping record production...  
  Topic: chuck_test 
  Record key: b375a193-73e0-44c2-9acc-6c916280ce7a 
  Payload: {"id":"b375a193-73e0-44c2-9acc-6c916280ce7a","arr":["7adb9836-ceac-4392-b370-2bd07a2ed0d5","f6177472-daa7-4d75-9d9a-a270b672a2e0","4ed906bf-8fd7-4516-ae3e-a1
149c22ebaf","95bec2b2-6633-467c-bb5f-118cbd0c3816","982dd939-3849-486f-ab00-58abb5434ac8"]}                                                                             

✔  Dry run: Skipping record production...  
  Topic: chuck_test2 
  Record key: 7adb9836-ceac-4392-b370-2bd07a2ed0d5 
  Payload: {"id":"7adb9836-ceac-4392-b370-2bd07a2ed0d5"}


✔  Dry run: Skipping record production...  
  Topic: chuck_test2 
  Record key: f6177472-daa7-4d75-9d9a-a270b672a2e0 
  Payload: {"id":"f6177472-daa7-4d75-9d9a-a270b672a2e0"}


✔  Dry run: Skipping record production...  
  Topic: chuck_test2 
  Record key: 4ed906bf-8fd7-4516-ae3e-a1149c22ebaf 
  Payload: {"id":"4ed906bf-8fd7-4516-ae3e-a1149c22ebaf"}


✔  Dry run: Skipping record production...  
  Topic: chuck_test2 
  Record key: 95bec2b2-6633-467c-bb5f-118cbd0c3816 
  Payload: {"id":"95bec2b2-6633-467c-bb5f-118cbd0c3816"}


✔  Dry run: Skipping record production...  
  Topic: chuck_test2 
  Record key: 982dd939-3849-486f-ab00-58abb5434ac8 
  Payload: {"id":"982dd939-3849-486f-ab00-58abb5434ac8"}


⚠  Stopping the data generator  

[bobbyiliev]

Just tested this out! That refactor is 🔥! It strips away the complexity of parsing the faker methods and args! Well done!

[chuck-alt-delete]

@bobbyiliev @sjwiesman passed tests! Looking forward to your reviews

[bobbyiliev]

Looks great! Well done with this!