feat(data list): rich task-grouped listing (task / size / records / format)#376
Merged
Conversation
`tracebloc data list` was a bare list of table names. It now groups datasets by modality (Image / Text / Tabular / Time-series) and shows, per dataset: split (train/test), record count with the right noun (images / documents / rows), real byte size, format (jpg/txt, or csv · N cols), class count, and how long ago it was ingested — plus a summary line and --all to reveal the framework tables (the ingest-run journal) hidden by default. Data sources (read-only, no backend round-trip): - push.ListDatasetsDetailed queries the mysql pod: an information_schema pass (create-time + columns), then a per-table UNION for data_intent / COUNT(*) / COUNT(DISTINCT label) / extension. Real datasets are tables with a data_id column; framework tables (no data_id) are flagged System. - Size comes from a du of the shared PVC on the jobs-manager pod (where files live). The DB size is metadata-only for file datasets and InnoDB-padded for tiny tables — both misleading — so it is deliberately not used. Best-effort: if the du is unreachable, size shows "—". Notes / limits (validated against a real 32-dataset cluster): - The specific task (image_classification vs object_detection, …) is NOT stored in the cluster DB, so grouping is by inferred modality (extension + time/ sequence columns), not the 16 exact tasks. Exact-task grouping needs the ingestor to persist the task. - Tabular/time-series show size "—": their data is DB rows, not PVC files. - "N classes" shows only when the label repeats (classes < records), so a continuous regression target isn't mislabelled as classes. --output-json emits per-dataset objects (modality/intent/records/classes/format/ size_bytes/ingested); --all includes system tables. Tests cover the parsers (schema/data/du), modality inference, humanBytes, the grouped render, --all, and the JSON shape. gofmt/vet/lint/deadcode/file-budget green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
|
@BugBot run |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Two Bugbot findings on #376: - Use the existing push.SharedRoot for the du path instead of a new SharedDataPath const, so a future mount-path change can't leave the size du pointed at a stale path while staging/teardown move. - datasetModality only keyed Image/Text off a populated extension, so an ingested-but-empty (0-row) file dataset — NULL extension — fell through to Tabular/"Other" with a misleading "csv · 0 cols". Now the Tabular branch requires records (an empty table's modality is genuinely unknowable → "Other"), and formatCell renders "—" for an undetermined modality instead of implying csv. Tests cover the empty-file case + the neutral format. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
|
@BugBot run |
Three Bugbot findings on #376: - HIGH: the per-table data pass hard-coded COUNT(DISTINCT label), but label is optional (self-supervised text tasks omit it) — one such table would fail the whole UNION ALL and exit 7 for the entire listing. Each per-table SELECT now references only the columns that table actually has (label / data_intent / extension), falling back to a constant otherwise. - Reuse exported push.HumanBytes instead of a private humanBytes that had drifted (%.1f vs %.2f, extra TiB), so data list matches ingest/over-cap sizes. - --output-json keeps `datasets` as the []string of names (the additive-only contract in docs/json-output.md forbids re-typing existing fields) and moves the rich per-dataset objects to a new `details` array; doc updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
|
@BugBot run |
…y-state polish Four Bugbot findings on #376: - Medium: relativeTime parsed the MySQL create_time wall clock as UTC, skewing every "Xh ago" by the server's tz offset. Now the schema pass also selects UNIX_TIMESTAMP(create_time) (a tz-safe epoch) and the "ago" math uses that; the ISO string is kept for display/JSON. - Medium: the size du treated any non-zero exit as total failure, blanking every dataset's size if one shared-PVC entry was unreadable (jobs-manager can hit EACCES). Appended `|| true` so readable entries still yield sizes. - Low: under --all, system tables rendered "0 B" (they aren't du-sized); now they show "—" like other unknown sizes. - Low: when every table is a system table and --all is off, the empty-state branch now still prints the "N system table(s) hidden — show with --all" hint. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
|
@BugBot run |
… lists Two Bugbot findings on #376: - Medium: datasetModality only matched "txt" for Text, but the ingestor accepts both .txt and .text (preflight accept-set / textExtensions). A .text dataset was grouped as "Other" with the wrong noun/format. Now matches txt + text. - Low: the schema query's GROUP_CONCAT of column names could truncate at MySQL's default group_concat_max_len (1024), under-counting feature columns / dropping modality markers on wide tables. Raise it to 1 MiB via a leading SET SESSION. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
|
@BugBot run |
Resolve internal/cli/data_list.go: keep the rich modality-grouped renderer (--all, per-dataset size/records/format/split, details JSON) and adopt #370's cleaner-output conventions on the shared surface — no banner, honest empty state via invokedName() + Para. Fold #370's reservedTables set into ListDatasetsDetailed's System-table detection so the ingest-run journal and ingest-meta store are hidden by default (alongside the existing data_id heuristic), consistent with how ListDatasets now filters them. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
|
@BugBot run |
Bugbot (commit 1cfdcc3): datasetRow pinned size to %9s and records to %-12s, but HumanBytes emits 10-char values ("100.00 KiB") and text counts reach "100 documents" (13). Go treats those widths as minimums, so a common large dataset printed its full value and pushed size / format / freshness right on that row alone — breaking the aligned table. Size the records and size columns to their widest cell, like name and format already are. Measure and pad by display width (runes) via padRight/padLeft, not fmt's byte-based %*s, so the multi-byte em dash and middot don't skew alignment either. Add TestRenderDataList_ ColumnsAlign asserting the format column starts at the same offset on a small row and a wide (100000-document / 100 KiB) row. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
|
@BugBot run |
Two follow-ups from Bugbot on 5c0b0b4: - Format-width cap without truncate (Low): fmtW was capped at 28 but format cells are never truncated, so a wide "csv · N cols · M classes" would overflow and shift freshness — reintroducing the overflow the dynamic sizing just fixed. Drop the cap: format cells are system-generated and bounded, so sizing to content stays aligned and keeps the full text. Names keep cap + truncate (user-controlled). - Unreachable "files" fallback (Medium premise): the base="files" arm in formatCell's Image/Text branch was dead — modality is extension-driven, so the extension is always set there. Move "files" to the reachable default branch, gated on a filename column, so a populated file table whose extension wasn't recorded reads "files" instead of "—" (it stays "Other" — image vs text is genuinely undeterminable without the extension). Extract hasCol and reuse it in datasetModality. Add TestFormatCell_FileWithoutExtension. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
|
@BugBot run |
9 tasks
…ame cap Three follow-ups from Bugbot on 8418436: - JSON ingested time lacked a timezone (Medium): the `ingested` field used a session-tz DATE_FORMAT string that looked like ISO-8601, while the human "ago" text used the UTC epoch — they disagree when the MySQL session tz isn't UTC. Make CreatedUnix (the UTC epoch) the sole time source and emit `ingested` as explicit UTC RFC3339 (Z-suffixed) via ingestedISO. Drop the naive DATE_FORMAT column + the CreatedAt field. - Listing failed if a table vanished (Medium): the data pass is one UNION ALL over the schema snapshot, so a table dropped by a concurrent `data delete` between the two queries failed the whole listing (exit 7). Retry the schema+data pair once — a fresh snapshot drops the vanished table and the rebuilt query succeeds; a broken mysql still fails the retry and surfaces the error. - System names capped without truncate (Low): the --all System section capped sysNameW at 24 but padRight never truncates, so a >24-char system name would overflow — the same false-cap class as the format column. Drop the cap; system names are framework-generated and short. Tests: schema fixtures move to the 3-column layout; add the vanished- table retry test and a UTC-explicit `ingested` assertion. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
|
@BugBot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 70b07e2. Configure here.
…asets
Read each dataset's task from the run journal (tracebloc_ingest_runs.task,
persisted by data-ingestors) and group by it, instead of only inferring
modality:
- list_detailed.go reads task via a guarded query — only when the journal
actually has the column (known from the schema pass), so a cluster whose
ingestor predates it is never queried for it. DatasetInfo gains Task.
- datasetModality is now authoritative when a task is recorded (task→modality
map), falling back to shape inference for datasets ingested before the
column shipped. Groups are labelled by the humanized task ("Image
classification"), ordered by modality family; JSON gains `task`.
Fill the size column for row-based datasets:
- fetch information_schema.data_length and use it as the size for tabular /
time-series datasets (whose data lives in the table, not the PVC). File
datasets keep the du-of-PVC size. The file-vs-row signal is the per-row
Extension, NOT a filename column — filename/extension/annotation are
framework columns on every table, so only a non-empty extension marks
staged files (caught against the live cluster: tabular tables carry a
filename column yet are row-based).
Verified against test0721 (32 datasets): pre-persistence datasets fall back
to inferred modality and now show a DB size instead of "—".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
|
@BugBot run |
Bugbot on 0832e30: an empty dataset (0 rows) has an empty extension too, so it fell into the row-based branch and took data_length — InnoDB's one-page (16 KiB) allocation, not real data — implying an empty, ⚠-flagged dataset holds a page of content. Only use DBBytes for row-based datasets that actually have rows (Extension == "" && Records > 0); leave an empty dataset sizeless ("—"), consistent with its empty flag. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
|
@BugBot run |
…istry Bugbot on 7f75028: taskModality + humanizeTask re-enumerated the ingest tasks, so section headers diverged from the rest of the CLI ("Time series classification" vs "Time-series classification", "Seq2seq" vs "Sequence-to-sequence") and would drift as the registry gains a task — exactly what category.go (cli#74) exists to prevent. Drop both and use the registry (push): - groupLabel → push.Lookup(task).Label for a known task (canonical, matches `data ingest`), the raw id for an unknown task, else the inferred modality. - datasetModality → push.IsImage/IsText/IsTabular for a known task. The registry files time-series under FamilyTabular, so a known time-series task now reports the Tabular family (its distinct label still heads its own group); the "Time-series" bucket remains only for the inference fallback (datasets ingested before the task was recorded). No behaviour change for pre-persistence datasets (verified against test0721: same Image/Text/Tabular/Time-series fallback groups). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Contributor
Author
|
@BugBot run |
…t's size applyDatasetSizes took any du hit as authoritative before checking Extension, so a row-based table with a dest dir on the shared PVC got that directory's size instead of its DBBytes — the documented file-vs-row rule never ran. Gate the du lookup on Extension (the file-bearing signal) and cover the stray-dir case in the test. Bugbot finding on #376. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
Author
|
bugbot run |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 00dceda. Configure here.
LukasWodka
changed the title
saadqbal
approved these changes
Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
tracebloc data listwas a bare bullet list of table names ("looks very empty").It now renders a rich, task-grouped table — each dataset's real task,
split (train/test), record count, size, format, classes, and freshness:
Per dataset: status · split · records (with the right noun) · size · format ·
classes · ingested. Datasets are grouped by their real task (the canonical
registry label — "Image classification", "Time-series classification", …),
ordered by family. Framework tables (the ingest-run journal) are hidden unless
--all.--output-jsonemits per-dataset objects. Validated against the realtest0721(32 datasets, every task).How (read-only, no backend round-trip)
push.ListDatasetsDetailedqueries the mysql pod: aninformation_schemapass(create-time,
data_length, columns), a per-tableUNION(data_intent,COUNT(*),COUNT(DISTINCT label), extension), and the ingest-run journalfor each dataset's task. SQL goes over stdin so backtick-quoted names need
no shell escaping; a one-shot retry survives a table dropped mid-listing.
tracebloc_ingest_runs.task— only when that columnexists (guarded), so it degrades gracefully. The group label and the modality
(family / record noun / format) both come from the CLI's category registry
(
push/category.go, the same sourcedata ingestuses), so they never driftas tasks are added. Datasets ingested before task-persistence fall back to
inferred modality (extension / column shape).
duof the shared PVC forfile datasets (images/text), and the table's
data_lengthforrow-based datasets (tabular / time-series). The file-vs-row signal is the
per-row extension —
filename/extensionare framework columns on everytable, so only a non-empty extension marks staged files. Empty datasets stay
sizeless (
—), matching their ⚠ flag.Addressed the two known limits from the first pass
The initial version could only infer modality and left row-based size blank.
Both are now solved:
it. feat(journal): record each run's task in the ingest-run journal data-ingestors#386 (merged) records each run's
taskin theingest-run journal; this PR reads it back and groups by it. Forward-looking:
newly ingested datasets show their exact task; datasets ingested before data ingest: strip surrounding quotes from the path prompt + clarify dataset-name rules #386
fall back to inferred modality until re-ingested.
data_lengthinstead of—, since their data lives in the table rather than on the PVC.Kept intentionally:
N classesshows only when the label repeats(
classes < records), so a continuous regression target isn't mislabelled.Tests
Parsers (schema / data / du / task), size selection (du vs
data_length, empty →—, stray-PVC-dir guard), registry-driven task label + family, modality-inferencefallback, column alignment (rune-width), the grouped render,
--all, and the JSONshape. gofmt / vet / staticcheck / deadcode / file-budget green; e2e (kind) green;
Bugbot clean.
🤖 Generated with Claude Code