Repository navigation
Releases: dbrownems/datagen
Release list
v0.8.61 - Sort-by-column constraint
Generated tables now satisfy the Tabular engine's sort-by-column constraint: when column D has SortByColumn=S, every D value pairs with exactly one S value. Eliminates 'Cannot order [D] by [S] because at least one value in [D] has multiple distinct values in [S]' refresh warnings.
- ColumnConfig.sort_by_column populated from VPAX metadata
- Spark generator builds paired pool (round-robin) aligned to the sort column
- Topo-sort ensures the sort column is processed before its dependent
- README: new 'Sort-by columns' section
v0.8.60
Histograms: reconcile pinned-row total vs target row_count.
- If pinned absolute-count rows sum > row_count, override row_count to the pinned sum (logged).
- If pinned absolute-count rows sum < row_count, generate the difference as filler rows (logged).
- No more ValueError on histogram total exceeding row_count.
v0.8.59
v0.8.58: Add optional <vpax>.histograms.yaml sidecar
Optional <vpax>.histograms.yaml sidecar to preserve table histograms across YAML regeneration.
- Place histograms in
<vpax-stem>.histograms.yamlnext to the main config (typically the lakehouseFiles/datagen/folder). generate()overlays the sidecar onto the config every run; sidecar wins per-table.- New CLI
python -m datagen extract-histograms <config>.yamlto migrate existing hand-edited histograms out of the main YAML.
v0.8.57: Add sequential date key styles for datetime PKs
Datetime key columns (relationship 'one' side or PK) now generate unique sequential dates, fixing 'duplicate value not allowed' refresh errors.
- New key_style values: sequential_date, sequential_month_end, sequential_month_start.
- Names like MonthEndDate / EOMDate / LastDayOfMonth detected as monthly-granularity keys (one row per month).
- Newest date anchored at start of current year (daily) or most recently completed month (monthly), walking back.
v0.8.56
generate() now accepts an array of DAX query strings as queries_path in addition to the existing JSONL trace path. Useful for seeding column values from a small handful of queries without writing a trace file.
Readme documents both forms and the minimum JSONL shape (eventClass=QueryEnd, cols.TextData=<DAX text>). All other JSONL fields are ignored.
v0.8.55
Strict argument validation in generate() (typos like mode="direclake" now raise ValueError immediately).
Direct Lake on OneLake fixes:
- Generated models are now properly tagged Direct Lake on OneLake (PBI_ProTooling, sourceLineageTag).
- Relationship type alignment runs directly on the BIM (PK-wins) so all original relationships survive deployment, instead of being silently dropped at TOM-deserialize time.
mode_labelis set on the deploy path so the reported mode can't disagree with what was actually deployed.
Readme refreshed.
v0.8.32
v0.8.31
- QueryRunnerStatus singleton for thread-safe live stats
- Connection summary table after ramp-up
- Periodic 60s progress reports with min/avg/max latency
- QueryRunnerLogger: lock-free dual console+file logging via BlockingCollection
- Async file write overlapped with synchronous Console.WriteLine
- Log file opened with FileShare.Read for live tailing
- All console output timestamped