Skip to content

Make datafusion-cli object store tracing mode work for local files #18119

@alamb

Description

@alamb

Is your feature request related to a problem or challenge?

@BlakeOrth and I have been working on an object store tracing mode which is great!

For exampele

DataFusion CLI v50.2.0
> \object_store_profiling trace
ObjectStore Profile mode set to Trace
> SELECT COUNT(*) from 'https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_1.parquet' where "SearchPhrase" <> '';
+----------+
| count(*) |
+----------+
| 131559   |
+----------+
1 row(s) fetched.
Elapsed 0.836 seconds.

Object Store Profiling
Instrumented Object Store: instrument_mode: Trace, inner: HttpStore
2025-10-17T09:24:25.641605+00:00 operation=Get duration=0.026670s size=8 range: bytes=174965036-174965043 path=hits_compatible/athena_partitioned/hits_1.parquet
2025-10-17T09:24:25.668363+00:00 operation=Get duration=0.027705s size=34322 range: bytes=174930714-174965035 path=hits_compatible/athena_partitioned/hits_1.parquet
2025-10-17T09:24:25.767319+00:00 operation=Get duration=0.039514s size=15503 range: bytes=5120273-5135775 path=hits_compatible/athena_partitioned/hits_1.parquet
2025-10-17T09:24:25.768711+00:00 operation=Get duration=0.085509s size=3895852 range: bytes=145483536-149379387 path=hits_compatible/athena_partitioned/hits_1.parquet
2025-10-17T09:24:25.817364+00:00 operation=Get duration=0.039122s size=61815 range: bytes=46392516-46454330 path=hits_compatible/athena_partitioned/hits_1.parquet

However, when I try to use it on a local file it doesn't work

Describe the solution you'd like

I would like it also to trace requests when reading local files too

For example, create a local file:

COPY 'https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_1.parquet' to 'hits_1_index.parquet';

And then run (note the lack of profiling):

DataFusion CLI v50.2.0
> \object_store_profiling trace
ObjectStore Profile mode set to Trace
> SELECT COUNT(*) from 'hits_1_index.parquet'  where "SearchPhrase" <> '';
+----------+
| count(*) |
+----------+
| 131559   |
+----------+
1 row(s) fetched.
Elapsed 0.225 seconds.

Object Store Profiling

Describe alternatives you've considered

No response

Additional context

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions