v0.1.0

v0.1.0 (2025-06-25)

Bug Fixes

• Add node_id to all nodes (f927dcd)

• Batch should be column, not parameters (73eeb9e)

• Condition for ignoring files during deletion (9da0e0f)

The manifest.json was being deleted by mistake.

• Correct name for placeholder models (fa07609)

feat: implement all missing edge models

• Ducklake integration using dev version for upcoming dbt-duckdb 1.4.1 (effe0d7)

• Duplicate alias for source_id and target_id columns (d6b6790)

• Ensure tags are checked out (d833c89)

• Generate sequential node ID globally for all nodes (8d019ac)

• Genre loading queries (abc6833)

refactor: reorganize models into stage and marts

feat: support for edge loading (untested)

• Genre nodes become a single table to ensure uniqueness (7ec7f03)

• Incorrect S3 secret variable (6fa7394)

• Missing description for playcount (c505c9c)

• Missing node ID dataset-based prefix (bfecf9f)

• Missing nodes prefix on ref table (58b04f5)

• Missing underscore after prefix (3a0d6d9)

• No longer defaulting to upstream dependencies (6d2d68a)

• Regression introduced by removing key_parts (668a31c)

• Removed extra bracket in log message (782bcc9)

• Should be alias, not name (d03e079)

• Should be list of list, not list of tuple (c3b7419)

• Sqlite prefix missing (78bae87)

• Switch to single table for genre nodes (6d5dd1f)

• Update graph loading process based on new config schema (eebc677)

• Update prune to use class prefix (49ca20f)

• Using map instead of list per node embedding (e6f1caf)

fix: add missing schema alter to add embedding property to all nodes

• Wrapper to copy from data mart via a temporary file (3cd0268)

• Wrong column name in schema (af2a693)

• Wrong filename case, should be RO, not ro (905a303)

• Wrong model name in schema (588f3bc)

• Wrong reference, missing schema prefix (701fb1d)

• Wrong variable order in log message (40cb055)

Chores

• Add description and pandas dep (b7c40d0)

• Add DUCKLAKE_PATH to .env (3ab2bee)

• Add kuzu as a dependency (f1e2a5c)

• Add S3 prefix for exports (88ee16c)

• Add solid background to diagram (7499f46)

• Add torch, torch-sparse, and torch-geometric deps (5ddd0f8)

• Better schema name organization for graphs (948759a)

• Click and minio deps (c6e450f)

• Default to eu-west-1, as MinIO also defaulted to it (6afa282)

• Delete example models (7012b5b)

• Fix version for python-semantic-release to match deps (3ff93d9)

• Github dark mode background color (ca97d44)

• Gitignore vscode directory (a2fcabd)

• Initial log message for export (1207e93)

• Initial log string is now a welcome string (0edefc6)

• Make sure we start from 0.1.0, not 1.0.0 (cb2b7c5)

• Remove unused dep (ba09c90)

• Remove unused deps and update docs referring to them (7da0d24)

• Replace with official GHA for python-semantic-release (7301029)

• Script to launch temporary docker container with KuzuDB Explorer for a database (52b617a)

• Setup dltctl CLI tool (replaces Makefile) (405d800)

• Simplify node and edge schemas, using Gremlin-like notation (887dddc)

• Solid background in individual rectangles (8f68204)

• Switch to a multi-database marts config (1cf615c)

• Temporarily removed (bd26438)

Schema was outdated and was blocking dbt run.

• Update config to match multi-database marts (d47f96f)

• Won't use the extra command in favor of one entry point (698a3d1)

• deps: Move python-semantic-release to dev deps (9df3ed6)

Documentation

• Add graph and shared (866b37c)

• Add specification for exports pruning (e42acbf)

• Dependency management development instructions (2f9393f)

• Duckdb init script description (94b1959)

• End-to-end documentation (190ef1b)

• Fix section links (e00a537)

• Latest.json is now manifest.json (6cdb057)

• Remove suffix from info boxes (170ebff)

• Requirements, quick start, architecture diagram (8009821)

• Schemas for nodes and edges of the music graph (409b4cb)

• Structured sections for README (5a6c128)

• Update schema for the DSN and MSDSL datasets (9a56aaa)

• Update storage layout (f60d021)

• Update storage layout (1383bfd)

• Update storage layout and specification for the ingest command (b66cffe)

• Using generic dark background (85476f1)

Features

• Add CLI args for read only and to reset (dbfefa8)

Container is now kept between sessions, unless explicitly reset.

• Backup restore can now specify source date (62a52af)

• Basic support for dbt run via dlctl transform (b3f6240)

• Catalog backup and restore (cb87830)

refactor: prefix is now set when instancing Storage

feat: Storage can now upload/download files or a directory

• Create directories for DuckDB databases (3976b19)

This way we can set the marts databases to be stored under local/marts/.

• Dbt debug option (f64313e)

• Debug option controlling log level (9e5f030)

• Directory structure for a DuckLake lakehouse (87ed579)

• Dlctl tools generate-init-sql (c24ca39)

This will output into local/init.sql. The scripts/init.example.sql or the gitignored scripts/init.sql are no longer used.

docs: add a help message to all commands

• Duckdb CLI init script to connect to the lakehouse (788691f)

• Error control for empty results (d5e1d65)

• Exception capture for KuzuOps (6a2a498)

• Export scripts that output parquet (218bfeb)

• Frp node embedding over KuzuDB (68a7514)

• Improve backups listing (8c5284d)

• Load all genres tables based on shared macro (ed8197b)

• Ls and prune for ingest and exports (e9a3505)

fix: add missing manifest to exports

fix: ignored file filtering

fix: add prefix logic to upload manifest

• Minio docker service (00f2dab)

• Node embedding computation and graph DB update (579b477)

• Option to use latest export when loading a graph (3b00439)

refactor: exports now stored using the same directory structure as marts

• Qol for CLI parameters and defaults, and logging (c6ee19b)

• Quality of life for explorer startup and exit (4562ab6)

• Replace export scripts with a load method from the new graph package (8c356d6)

• Replace load_dotenv with proper validation via environs (aacb1a3)

refactor: centralize storage and environment variable loading into shared packages

refactor: improve function naming and arguments

feat: set placeholder upload as optional

refactor: rename lastest.json to manifest.json

feat: storage now implements an env var loader with latest file paths

• Schema name without the 'main_' prefix (37b0bff)

• Setup semantic releases (ef856fd)

• Strip schema name from table name (fc83dfa)

• Stubs for node computation embedding command (f220fc2)

• Support file downloading from object storage (eff8a8b)

• Support for kaggle and hugging face ingestion (39f8fb5)

• Support for manual dataset ingestion (3b6c3d1)

• Support for running a subset of models during transform (4986aee)

Performance Improvements

• Switch to UNPIVOT strategy (51059e2)

Refactoring

• Better naming scheme for graph schemas, and node and edge tables (1716d17)

• Cleanup file to avoid inline comments (498534d)

• Con is now conn (cc99988)

• Embedding batch updates now handled directly by NodeEmbedder (5dc6e51)

• Genres/nodes and edges are now stored in the graphs mart (9963909)

docs: schemas updated with node and edge information and basic testing

• Graph manager is now ops (c84dbf5)

• Improved docs and better naming for DuckLake DBs (0b8097d)

• Latest export is now default, but re-exporting can be forced (54324c3)

• Log exception message without stack trace (1d52f40)

• Name source and target columns (b0c83fd)

feat: cast node IDs to integer

• Nodes and edges directories to match graph DB loading format (86ef29e)

feat: million song dataset, spotify and lastfm transformation

feat: improve deezer genres and edges mart table schemas

• Overall simplification of the explore graph script (94b0bb0)

• Qol, log message in lower case after colon (76cccc8)

• Qol, log message now includes epoch (51c96cd)

• Remove source from edges (ca15947)

• Remove uneeded echos (a5f86a0)

• Rename models to include a schema prefix (dd263ab)

feat: implement missing node models

• Rename music graph back to music_taste (d0d7a57)

• S3 access key and secret renamed to reflect common naming schema (d00a4c9)

• Table materialization is now default (35dc856)

• Taking advantage of parents accessor (1038678)

• Tools and utils moved to shared (3900df1)

feat: init SQL can now be returned as a string

fix: lakehouse relied on an init script that's no longer there

Using generate_init_sql to produce a string with the required SQL instead.

chore: uncommended code that didn't run due to KuzuDB bug

• Util is now templates for clarity (7518c06)

chore: groups no longer invokable without arguments

This had been added for better performance, but did nothing.

refactor: split export into standalone feature

Extracted from graph load and integrated into the existing export command (renamed from exports to export).

Testing

• Column only contains positive integers (c4459fb)

• List/array not null or empty (3ff573f)

• Make sure node IDs are globally unique (5d9a1ff)