Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache Field #316

Closed
lmath opened this issue Nov 2, 2022 · 0 comments
Closed

Cache Field #316

lmath opened this issue Nov 2, 2022 · 0 comments
Milestone

Comments

@lmath
Copy link
Contributor

lmath commented Nov 2, 2022

To load into big query, we need to take some data that is in a json following some particular schema, and transform it into something suitable to be understood by big query. Each time we were:

  1. looking up the schema associated with the schema key associated with the data
  2. converting the schema to a Field

We expect that for a given schema key, that its corresponding Field remains the same unless the schema has changed. So, this field can be cached, as long as we have a means to ensure that if the schema is updated, so is the field in a timely manner.

To ensure that, we add a TTL on the cache, taking the same approach as described here: snowplow/snowplow-rdb-loader#1086

where for our Field cache,
key = (SchemaKey, TTL)
value = Field

lmath added a commit that referenced this issue Nov 2, 2022
lmath added a commit that referenced this issue Nov 7, 2022
lmath added a commit that referenced this issue Nov 22, 2022
lmath added a commit that referenced this issue Nov 24, 2022
lmath added a commit that referenced this issue Nov 24, 2022
lmath added a commit that referenced this issue Nov 28, 2022
Thanks Dilyan for introducing an in-memory registry in CachedFieldSpec
Co-authored-by: Dilyan Damyanov <dilyand@gmail.com>
lmath added a commit that referenced this issue Nov 28, 2022
Thanks Dilyan for introducing an in-memory registry in CachedFieldSpec

Co-authored-by: Dilyan Damyanov <dilyand@gmail.com>
lmath added a commit that referenced this issue Nov 28, 2022
Thanks Dilyan for introducing an in-memory registry in CachedFieldSpec

Co-authored-by: Dilyan Damyanov <dilyand@gmail.com>
@lmath lmath added this to the 1.5.3 milestone Nov 28, 2022
@lmath lmath closed this as completed in 3ed5238 Nov 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant