0.15.0
Important new stuff:
MetaPlane
See meta-plane.md for motivation
- Introduced
MetaPlane/TableMeta/TransformMetainterfaces to decouple
metadata management from the compute plane - Added SQL reference implementation (
SQLMetaPlane,SQLTableMeta,
SQLTransformMeta) and rewiredDataStore,DataTable, and batch transform
steps to consume the new meta plane API - Added meta-plane design doc and removed legacy
MetaTableplumbing in lints,
migrations, and tests
InputSpec and key mapping
See key-mapping.md for motivation
- Renamed
JoinSpectoInputSpec - Added
keysparameter toInputSpecandComputeInputto support
joining tables with different key names - Added
OutputSpecandComputeOutput.keysto explicitly map transform keys
to output table primary keys - Fixed batch transform cleanup for aliased output keys and incomplete transform
keys
Step name overrides and uniform hash-based naming
- Extracted
make_mungled_step_name(cls, base_name, input_dts, output_dts)as a
public helper incompute.py; it encodes the step class, function name, and
table names into a short shake-128 hash suffix (e.g.my_func_9762dd6bae) ComputeStep.nameis now a plain stored attribute instead of a computed
property, so the name is fixed at construction time and readable without
re-hashing- All
PipelineSteptypes now accept an optionalname: str | Noneparameter;
when provided it overrides the auto-generated hash name, making it easy to
pin a stable name for a step independent of its inputs/outputs DatatableTransformandUpdateExternalTablewere previously using plain
names (e.g.update_item); they now usemake_mungled_step_namefor
consistency with the batch step typespipeline_input_to_compute_input()extracted fromBatchTransforminto a
module-level helper incompute.pyand reused byDatatableBatchTransformDatatableBatchTransform.inputsnow acceptsPipelineInput(same as
BatchTransform), enablingRequired/InputSpecwrappersbuild_compute()now raises immediately on duplicate step names
Python3.9 support is deprecated
Improvements and fixes
- Fixed dtypes mapping for TableStoreExcel, TableStoreJsonLine
- Fixed meta changes compute logic for
Requiredtables