Optimize memory usage with chunked seeder generation and benchmarking

Summary

Fixes issues #15 and #11, adds a user-configurable chunk size for seed generation, and replaces manual CLI progress output with CodeIgniter's native progress bar. Builds on the earlier memory/performance rework of SeederGenerator.php (streaming generator + keyset pagination), whose benchmark results are included below for reference.

Changes

1. Fix #15 — missing default value in migrations

MigrationGenerator.php:186-196 now emits 'default' => ... in the field definition:

• Numeric defaults emitted unquoted ('default' => 10,); strings via var_export ('default' => 'active',)
• Nullable columns with no explicit default get 'default' => null,
• auto_increment columns get no default; current_timestamp() keeps its existing raw-string path

2. Fix #11 — FileHandler path

HEAD already used APPPATH, so the literal fix was in place — but FileHandler.php:74-84 was hardened further: the path now uses DIRECTORY_SEPARATOR and creates Database/Migrations/ if it doesn't exist. Previously, file_put_contents silently failed into the CLI::error branch when the folder was missing — the likely remaining pain point in renamed-app setups.

3. Configurable chunk size (default 1000)

• SeederGenerator.php:24-29 — constructor now takes int $chunkSize = self::DEFAULT_CHUNK_SIZE (clamped to ≥1); all pagination paths use $this->chunkSize
• GetSeedCommand.php — new --chunk option with validation, plus $arguments/$options metadata so spark help get:seed documents it

 
php spark get:seed users --chunk 5000
 

4. CLI progress bar with percent

SeederGenerator.php:94-104 now uses CodeIgniter's native CLI::showProgress($processed, $totalRows), rendering [####......] 40% Complete in place — updated once per chunk and cleared with showProgress(false) when done, followed by a green summary line with the row count.

Test Results

 
tests/test_seeder.php              16/16 PASS  (incl. custom chunk size honored,
                                                output identical across chunk sizes,
                                                progress bar called per-chunk)
tests/test_migration_defaults.php   6/6  PASS  (string/int/decimal/null defaults,
                                                auto_increment excluded, timestamp raw)
 

Note: the migration defaults test drives generateField() with faked DESCRIBE rows, so it covers MySQL-shaped output — worth a quick live spark get:migration against a real DB to confirm end-to-end.

---

Background: Seeder Memory/Performance Rework

What changed in SeederGenerator.php

• True streaming via PHP generator — yieldTableRows() yields one row at a time while fetching from the DB in 1000-row chunks, so only one chunk ever lives in memory (SeederGenerator.php:180).
• Keyset pagination — when the table has a single-column primary key, chunks are fetched with WHERE pk > last ORDER BY pk instead of LIMIT/OFFSET (SeederGenerator.php:198). OFFSET re-scans skipped rows, making it O(n²) overall — the benchmark showed 64s vs 6.5s at 500k rows. Tables without a usable PK fall back to offset pagination automatically.
• Single file handle + buffered writes — the initial version called file_put_contents(..., FILE_APPEND) per row (open/lock/close × N). Now the file is opened once with fopen and rows are buffered and flushed once per chunk (SeederGenerator.php:63).
• Throttled progress output — progress prints once per chunk instead of once per row (2500 rows → 5 CLI writes, not 2500).

Benchmark Results

Ran via tests/bench_run.php (SQLite + CI4 stubs in tests/stubs.php, fresh process per run for clean peak-memory numbers). "Legacy" = the original all-rows-at-once approach.

Rows	Legacy time	Generator time	Legacy peak mem	Generator peak mem
10k	0.11s	0.11s	14 MB	4 MB
100k	1.32s	1.27s	148 MB	4 MB
500k	18.8s (needed 2G limit)	6.5s	728 MB	4 MB
1M	💥 OOM at default 512M	10.7s	fatal error	4 MB

Key takeaways:

• Memory is flat at 4 MB regardless of table size — the whole point of the generator. Legacy scales linearly and dies at ~700k rows on a default memory_limit.
• Speed is equal or better — at 500k rows the generator is ~3× faster because the legacy version thrashes memory concatenating a 176 MB string.
• The no-PK fallback (offset pagination) still holds 4 MB but takes 64s at 500k rows — that's why keyset pagination was worth adding.

Tests — tests/test_seeder.php

Verifies: file created, empty/migrations tables skipped, output passes php -l, row count and data integrity (quotes, nulls, first/last rows) across multi-chunk boundaries, and CLI output throttling. Run with:

php tests/test_seeder.php

test_performance.php in the repo root (the earlier yield-vs-array experiment) is now superseded by the real benchmark in tests/ — consider deleting it before committing.