Skip to content

Releases: dbrownems/datagen

v0.8.61 - Sort-by-column constraint

Choose a tag to compare

@dbrownems dbrownems released this 02 Jun 19:14

Generated tables now satisfy the Tabular engine's sort-by-column constraint: when column D has SortByColumn=S, every D value pairs with exactly one S value. Eliminates 'Cannot order [D] by [S] because at least one value in [D] has multiple distinct values in [S]' refresh warnings.

  • ColumnConfig.sort_by_column populated from VPAX metadata
  • Spark generator builds paired pool (round-robin) aligned to the sort column
  • Topo-sort ensures the sort column is processed before its dependent
  • README: new 'Sort-by columns' section

v0.8.60

Choose a tag to compare

@dbrownems dbrownems released this 07 May 20:16

Histograms: reconcile pinned-row total vs target row_count.

  • If pinned absolute-count rows sum > row_count, override row_count to the pinned sum (logged).
  • If pinned absolute-count rows sum < row_count, generate the difference as filler rows (logged).
  • No more ValueError on histogram total exceeding row_count.

v0.8.59

Choose a tag to compare

@dbrownems dbrownems released this 07 May 19:33

extract-histograms: ASCII-safe console output (replace Unicode arrow with '->' so the CLI works on default Windows consoles without PYTHONIOENCODING=utf-8).

v0.8.58: Add optional <vpax>.histograms.yaml sidecar

Choose a tag to compare

@dbrownems dbrownems released this 07 May 19:23

Optional <vpax>.histograms.yaml sidecar to preserve table histograms across YAML regeneration.

  • Place histograms in <vpax-stem>.histograms.yaml next to the main config (typically the lakehouse Files/datagen/ folder).
  • generate() overlays the sidecar onto the config every run; sidecar wins per-table.
  • New CLI python -m datagen extract-histograms <config>.yaml to migrate existing hand-edited histograms out of the main YAML.

v0.8.57: Add sequential date key styles for datetime PKs

Choose a tag to compare

@dbrownems dbrownems released this 07 May 19:17

Datetime key columns (relationship 'one' side or PK) now generate unique sequential dates, fixing 'duplicate value not allowed' refresh errors.

  • New key_style values: sequential_date, sequential_month_end, sequential_month_start.
  • Names like MonthEndDate / EOMDate / LastDayOfMonth detected as monthly-granularity keys (one row per month).
  • Newest date anchored at start of current year (daily) or most recently completed month (monthly), walking back.

v0.8.56

Choose a tag to compare

@dbrownems dbrownems released this 06 May 23:04

generate() now accepts an array of DAX query strings as queries_path in addition to the existing JSONL trace path. Useful for seeding column values from a small handful of queries without writing a trace file.

Readme documents both forms and the minimum JSONL shape (eventClass=QueryEnd, cols.TextData=<DAX text>). All other JSONL fields are ignored.

v0.8.55

Choose a tag to compare

@dbrownems dbrownems released this 06 May 22:25

Strict argument validation in generate() (typos like mode="direclake" now raise ValueError immediately).

Direct Lake on OneLake fixes:

  • Generated models are now properly tagged Direct Lake on OneLake (PBI_ProTooling, sourceLineageTag).
  • Relationship type alignment runs directly on the BIM (PK-wins) so all original relationships survive deployment, instead of being silently dropped at TOM-deserialize time.
  • mode_label is set on the deploy path so the reported mode can't disagree with what was actually deployed.

Readme refreshed.

v0.8.32

Choose a tag to compare

@dbrownems dbrownems released this 17 Apr 18:08

Add Datagen Main notebook to repo (notebook/Datagen Main.Notebook/) and improve README with Requirements section, DaxLoadGen companion link, and updated build-from-source instructions.

v0.8.31

Choose a tag to compare

@dbrownems dbrownems released this 12 Apr 23:12
  • QueryRunnerStatus singleton for thread-safe live stats
  • Connection summary table after ramp-up
  • Periodic 60s progress reports with min/avg/max latency
  • QueryRunnerLogger: lock-free dual console+file logging via BlockingCollection
  • Async file write overlapped with synchronous Console.WriteLine
  • Log file opened with FileShare.Read for live tailing
  • All console output timestamped

v0.8.30

Choose a tag to compare

@dbrownems dbrownems released this 12 Apr 22:18
  • Set AdomdCommand.CommandTimeout=0 (no query timeout)
  • Add LoadGen CLI with Azure.Identity auth support
  • Single-file self-contained LoadGen.exe publish