Implementing User Defined Types and Custom Metadata in DataFusion - Apache DataFusion Blog

[giscus[bot]]

Implementing User Defined Types and Custom Metadata in DataFusion - Apache DataFusion Blog

https://datafusion.apache.org/blog/2025/09/21/custom-types-using-metadata/

[fbx31]

Hello, thanks a lot for this article, helped me a lot but unfortunately, I am still failing to obtain the field metadata. inside a dataframe.

I implemented an async udf which works fine and I added the return_field_from_args function as you mentioned.
fn return_field_from_args(&self, args: ReturnFieldArgs) -> datafusion::error::Result {
let field_name = INDEXED_VALUE.to_string() + args.arg_fields.get(0).map_or_else(|| "", |f| f.name());

    Ok(Arc::new(
        Field::new(field_name, DataType::new_fixed_size_list(DataType::Float32, self.dims as i32,true), true).with_metadata(
            [("OSM:index".to_string(), self.name.to_string())]
                .into_iter()
                .collect(),
        ),
    ))
}

I call df.with_column(alias, my_func_call).create_physical_plan().await then execute_stream on this plan.
When I print schema with debug (:?), I get all fields but NO metadata attached to the field I created.

Did I miss something ? Is with_colum erasing my metadata ?

BTW, a similar article explaining how to add medata to a dataframe's schema could be very helpfull as well. I am struggling with this use case as well.

Thanks for any help