-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Closed
Labels
enhancementNew feature or requestNew feature or request
Description
Is your feature request related to a problem or challenge?
@BlakeOrth and I have been working on an object store tracing mode which is great!
For exampele
DataFusion CLI v50.2.0
> \object_store_profiling trace
ObjectStore Profile mode set to Trace
> SELECT COUNT(*) from 'https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_1.parquet' where "SearchPhrase" <> '';
+----------+
| count(*) |
+----------+
| 131559 |
+----------+
1 row(s) fetched.
Elapsed 0.836 seconds.
Object Store Profiling
Instrumented Object Store: instrument_mode: Trace, inner: HttpStore
2025-10-17T09:24:25.641605+00:00 operation=Get duration=0.026670s size=8 range: bytes=174965036-174965043 path=hits_compatible/athena_partitioned/hits_1.parquet
2025-10-17T09:24:25.668363+00:00 operation=Get duration=0.027705s size=34322 range: bytes=174930714-174965035 path=hits_compatible/athena_partitioned/hits_1.parquet
2025-10-17T09:24:25.767319+00:00 operation=Get duration=0.039514s size=15503 range: bytes=5120273-5135775 path=hits_compatible/athena_partitioned/hits_1.parquet
2025-10-17T09:24:25.768711+00:00 operation=Get duration=0.085509s size=3895852 range: bytes=145483536-149379387 path=hits_compatible/athena_partitioned/hits_1.parquet
2025-10-17T09:24:25.817364+00:00 operation=Get duration=0.039122s size=61815 range: bytes=46392516-46454330 path=hits_compatible/athena_partitioned/hits_1.parquet
However, when I try to use it on a local file it doesn't work
Describe the solution you'd like
I would like it also to trace requests when reading local files too
For example, create a local file:
COPY 'https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_1.parquet' to 'hits_1_index.parquet';And then run (note the lack of profiling):
DataFusion CLI v50.2.0
> \object_store_profiling trace
ObjectStore Profile mode set to Trace
> SELECT COUNT(*) from 'hits_1_index.parquet' where "SearchPhrase" <> '';
+----------+
| count(*) |
+----------+
| 131559 |
+----------+
1 row(s) fetched.
Elapsed 0.225 seconds.
Object Store ProfilingDescribe alternatives you've considered
No response
Additional context
No response
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request