New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cache Field #316
Milestone
Comments
lmath
added a commit
that referenced
this issue
Nov 28, 2022
Thanks Dilyan for introducing an in-memory registry in CachedFieldSpec Co-authored-by: Dilyan Damyanov <dilyand@gmail.com>
lmath
added a commit
that referenced
this issue
Nov 28, 2022
Thanks Dilyan for introducing an in-memory registry in CachedFieldSpec Co-authored-by: Dilyan Damyanov <dilyand@gmail.com>
lmath
added a commit
that referenced
this issue
Nov 28, 2022
Thanks Dilyan for introducing an in-memory registry in CachedFieldSpec Co-authored-by: Dilyan Damyanov <dilyand@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
To load into big query, we need to take some data that is in a json following some particular schema, and transform it into something suitable to be understood by big query. Each time we were:
We expect that for a given schema key, that its corresponding Field remains the same unless the schema has changed. So, this field can be cached, as long as we have a means to ensure that if the schema is updated, so is the field in a timely manner.
To ensure that, we add a TTL on the cache, taking the same approach as described here: snowplow/snowplow-rdb-loader#1086
where for our Field cache,
key =
(SchemaKey, TTL)
value =
Field
The text was updated successfully, but these errors were encountered: