-
Notifications
You must be signed in to change notification settings - Fork 4k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
document how to set stream schemas in CDK
- Loading branch information
1 parent
636a21e
commit bf1ac1e
Showing
1 changed file
with
24 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
# Defining your stream schemas | ||
Your connector must describe the schema of each stream it can output using [JSONSchema](https://json-schema.org). | ||
|
||
The simplest way to do this is to describe the schema of your streams using one `.json` file per stream. You can also dynamically generate the schema of your stream in code, or you can combine both approaches: start with a `.json` file and dynamically add properties to it. | ||
|
||
The schema of a stream is the return value of `Stream.get_json_schema`. | ||
|
||
## Static schemas | ||
By default, `Stream.get_json_schema` reads a `.json` file in the `schemas/` directory whose name is equal to the value of the `Stream.name` property. In turn `Stream.name` by default returns the name of the class in snake case. Therefore, if you have a class `class EmployeeBenefits(HttpStream)` the default behavior will look for a file called `schemas/employee_benefits.json`. You can override any of these behaviors as you need. | ||
|
||
Important note: any objects referenced via `$ref` should be placed in the `shared/` directory in their own `.json` files. | ||
|
||
## Dynamic schemas | ||
If you'd rather define your schema in code, override `Stream.get_json_schema` in your stream class to return a `dict` describing the schema using [JSONSchema](https://json-schema.org). | ||
|
||
## Dynamically modifying static schemas | ||
Place a `.json` file in the `schemas` folder containing the basic schema like described in the static schemas section. | ||
Then, override `Stream.get_json_schema` to run the default behavior, edit the returned value, then return the edited value: | ||
``` | ||
def get_json_schema(self): | ||
schema = super().get_json_schema() | ||
schema['dynamically_determined_property'] = "property" | ||
return schema | ||
``` |