with_column_renamed is always a NO OP when table name is ?table?

[RougeEtoile]

let test_df = ctx.read_csv("test.csv",  CsvReadOptions::new()).await?;
let test_df = test_df.with_column_renamed("id", "renamedID")?;
println!("{:#?}", test_df.explain(true, true)?);

+---------------+-----------------------------------------------------------------------------------------------------------------+
| plan_type     | plan                                                                                                            |
+---------------+-----------------------------------------------------------------------------------------------------------------+
| logical_plan  | TableScan: ?table? projection=[id, name]                                                                        |
| physical_plan | CsvExec: files=[<>/test.csv], has_header=true, limit=None, projection=[id, name] |
|               |                                                                                                                 |
+---------------+-----------------------------------------------------------------------------------------------------------------+

This time using ?table? as a qualifier

let test_df = ctx.read_csv("test.csv",  CsvReadOptions::new()).await?;
let test_df = test_df.with_column_renamed("?table?.id", "renamedID")?;
test_df.explain(false, false)?.show().await?;

Error: SchemaError(FieldNotFound { field: Column { relation: None, name: "?table?.id" }, valid_fields: Some([Column { relation: Some("?table?"), name: "id" }, Column { relation: Some("?table?"), name: "name" }]) })

This time registering table so it has a set table name

let test_df = ctx.read_csv("test.csv",  CsvReadOptions::new()).await?;
ctx.register_table("t1", test_df)?;
let test_df = ctx.table("t1")?;
let test_df = test_df.with_column_renamed("id", "renamedID")?;
test_df.explain(false, false)?.show().await?;

+---------------+-------------------------------------------------------------------------------------------------------------------+
| plan_type     | plan                                                                                                              |
+---------------+-------------------------------------------------------------------------------------------------------------------+
| logical_plan  | Projection: ?table?.id, ?table?.name, alias=t1                                                                    |
|               |   TableScan: ?table? projection=[id, name]                                                                        |
| physical_plan | ProjectionExec: expr=[id@0 as id, name@1 as name]                                                                 |
|               |   CsvExec: files=[<>/test.csv], has_header=true, limit=None, projection=[id, name] |
|               |                                                                                                                   |
+---------------+-------------------------------------------------------------------------------------------------------------------+

This time registering table so it has a set table name and using that table name as a qualifier

let test_df = ctx.read_csv("test.csv",  CsvReadOptions::new()).await?;
ctx.register_table("t1", test_df)?;
let test_df = ctx.table("t1")?;
let test_df = test_df.with_column_renamed("t1.id", "renamedID")?;
test_df.explain(false, false)?.show().await?;

+---------------+---------------------------------------------------------------------------------------------------------------------+
| plan_type     | plan                                                                                                                |
+---------------+---------------------------------------------------------------------------------------------------------------------+
| logical_plan  | Projection: t1.id AS renamedId, t1.name                                                                             |
|               |   Projection: ?table?.id, ?table?.name, alias=t1                                                                    |
|               |     TableScan: ?table? projection=[id, name]                                                                        |
| physical_plan | ProjectionExec: expr=[id@0 as renamedId, name@1 as name]                                                            |
|               |   ProjectionExec: expr=[id@0 as id, name@1 as name]                                                                 |
|               |     CsvExec: files=[<>/test.csv], has_header=true, limit=None, projection=[id, name] |
|               |                                                                                                                     |
+---------------+---------------------------------------------------------------------------------------------------------------------+

I have two lines of thinking on this

1. with_column_renamed seems to be the only function in the DataFrame API that always requires the use of the fully qualified column name, this should be changed to use the unqualified column name.
2. Why are tables being set to ?table? ? I assume this was chosen because it isn't valid sql but why not always generate a valid table name from functions like ctx.read_csv(). The current implementation is awkward because using ?table? as a qualifier returns an error.

[Jefffrey]

I agree with point 1, in that it should be able to infer the full qualified column from an unqualified column name, as other methods in the Dataframe API are able to, though will need to enforce ambiguity checks to catch any ambiguous references.