Skip to content

data list: text datasets show no size (du measures disk blocks, and 0 is indistinguishable from unknown) #491

Description

@shujaatTracebloc

What happens

tb data list shows no size for every text dataset — text classification, token
classification, masked/causal language modeling, seq2seq, sentence-pair, embeddings — while
image datasets in the same namespace show real sizes:

Image classification · 2
  image_train   train  6 images       180.00 KiB  jpeg · 2 classes
Masked language modeling · 2
  mlm_train     train  5 documents            —  txt
Text classification · 2
  text_train    train  45 documents           —  txt · 3 classes

Record counts, classes, extension and task are all correct — only the size is missing.

Why

Two independent defects compound.

1. The size is mis-measured. Sizes come from du -sk <shared>/* run inside the jobs-manager
pod. du without --apparent-size measures st_blocks, and on a bind-mounted host filesystem
(Docker Desktop, and any mount that doesn't report block allocation for small files) that returns
0 for a directory of small files. Observed directly on the node:

0     <shared>/mlm_train
0     <shared>/text_train
180   <shared>/image_train

Image datasets survive only because their files are large enough to occupy reported blocks. Text
datasets are dozens of small .txt documents, so every one of them measures zero.

2. Zero and "unknown" are the same value. sizeCell renders on SizeBytes > 0, so a dataset
measured at 0 is indistinguishable from one the CLI never managed to measure. That is why the
symptom presents as "the size is missing" rather than "the size is wrong" — and why it looks like a
broken lookup when the lookup is fine.

Tabular/time-series datasets hide defect 1 by accident: their Extension is empty, so they take
the DBBytes branch instead of du.

Fix

  • Measure apparent size (st_size), probing --apparent-size support once, since busybox du
    rejects the flag. Not a du --apparent-size … || du … chain: du exits non-zero if any path
    is unreadable, so that would silently fall back to block sizes intermittently.
  • Track whether a size was measured separately from its value, so a real zero renders 0 B
    and only a failed lookup renders .

Metadata

Metadata

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions