Skip to content

Tolerate null created/modified in searchable extract - #327

Merged
jp-spotter merged 1 commit into
thoughtspot:devfrom
jp-spotter:fix/searchable-dependent-keyerror
Jul 23, 2026
Merged

Tolerate null created/modified in searchable extract#327
jp-spotter merged 1 commit into
thoughtspot:devfrom
jp-spotter:fix/searchable-dependent-keyerror

Conversation

@jp-spotter

Copy link
Copy Markdown
Collaborator

Problem

The searchable metadata extract crashes mid-run (KeyError: 'created') when the ThoughtSpot API
omits a timestamp for certain object types — observed: FEEDBACK dependents omit created,
LOCAL_GROUP groups omit modified. One missing field aborts the whole extract, so affected
clusters get no data at all.

Fix

Treat created/modified as a nullable class in the searchable models, and coerce through a
shared helper that tolerates absent/null:

  • validators.utc_from_millis() — epoch-millis → UTC datetime, or None when absent (guards the
    /1000 against KeyError/TypeError).
  • validators.ensure_datetime_is_utc now short-circuits None, so model validators inherit it.
  • created/modifiedOptional on User, Group, Tag, DataSource, MetadataObject,
    DependentObject; transformers use src.get(...) via the helper.

A batch extract should capture whatever is available (null included) rather than crash.

No contract change

  • DB columns were already nullable (SQLAlchemy default) — no schema migration needed.
  • Genuinely-required timestamps in other tools still reject None loudly (their fields stay
    non-optional; pydantic rejects at the type check).
  • Return shape unchanged.

Validation

  • New unit tests (tests/test_searchable_transformer.py); full suite green.
  • Audited the live transformers against two clusters (~140k records) — 0 failures, including
    the LOCAL_GROUP/modified case that previously failed.

Scope

Searchable only. The shared validators change enables the same treatment for the other tools'
transformers (archiver, scriptability, user-management, …) as a follow-up.

@jp-spotter
jp-spotter requested a review from a team as a code owner July 23, 2026 19:21
@jp-spotter
jp-spotter merged commit 766a998 into thoughtspot:dev Jul 23, 2026
3 checks passed
@jp-spotter
jp-spotter deleted the fix/searchable-dependent-keyerror branch July 24, 2026 05:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant