CI workflow + @materialize decorator - #1
Conversation
- @materialize(ttl=, version=, key=, key_fn=, background=): hit returns a frame without calling the function; miss runs it, serves the result immediately, persists write-behind - MaterializedFrame: .arrow()/.df()/.to_pylist()/len(), .sql() runs server-side against the entry ("this" names the cached data) - Fail-open: registry/persist failures degrade to running uncached - EntryStore.query_table() extracted for frame SQL; README leads with the decorator
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| def sql(self, sql: str) -> pa.Table: | ||
| """Run SQL server-side against this entry's database; the identifier | ||
| `this` names the cached data (e.g. "SELECT x FROM this LIMIT 5").""" | ||
| return self._store.query_table(self.entry, _THIS.sub(DATA_TABLE, sql)) |
There was a problem hiding this comment.
nit: _THIS.sub(...) rewrites every standalone this token in the SQL string, including ones inside string literals and quoted identifiers. frame.sql("SELECT channel FROM this WHERE label = 'this'") silently becomes ... WHERE label = 'data', returning wrong rows with no error. The documented FROM this path is fine, but the collision produces silent incorrect results. Consider a less collision-prone sentinel (e.g. a token unlikely to appear in real SQL) or scoping the substitution to table-reference positions. (not blocking)
| exc_info=True, | ||
| ) | ||
| pending = RegistryEntry(fingerprint=fingerprint, key=label) | ||
| return MaterializedFrame(store, pending, table=table, cached=False) |
There was a problem hiding this comment.
super nit: on a miss in the default (background) mode, pending has database_id=None, so calling .sql() on the returned frame raises StoreError("... no database yet") until the write-behind persist finishes. .arrow()/.to_pylist()/len() all work on a miss frame (they use the local table), so .sql() being the one method that doesn't is an easy surprise. Worth a note in the README/docstring that server-side .sql() requires a persisted entry (a hit, or background=False). (not blocking)
Decorator-first docs and demo; fix review nits from #1
1 participant
What
Two additions:
.github/workflows/ci.yml): flake8 + mypy + pytest on Python 3.11/3.12. Verified locally that the suite passes against the publishedhotdata0.8.0 from PyPI.@materializedecorator andMaterializedFrame(step 2 first cut):frame.sql("... FROM this ...")runs server-side against the entry's database.Tests
67 passing (9 new decorator tests: miss/hit, argument and version busting, TTL expiry, fail-open, server-side SQL, capture normalization). mypy and flake8 clean under the SDK-matching config.