load_table_from_dataframe column values loaded as null to BQ if column name has spaces in it

[osmanamjad]

Environment details

• OS type and version: Linux
• Python version: 3.10.10
• pip version: 22.3.1
• google-cloud-bigquery version: 3.10.0

Steps to reproduce

1. Create a Pandas dataframe with a column name that has a space in it.
2. Call load_table_from_dataframe on the pandas dataframe.
3. Resulting BQ table does not have values for the column whose name had a space in it.

Code example

import pandas as pd
from google.cloud import bigquery

# Initialize a pandas DataFrame (Step 1)
data = {'Full Name': ['John', 'Alice', 'Bob'],
        'Age': [25, 30, 35],
        'City': ['New York', 'London', 'Paris']}
df = pd.DataFrame(data)

# Create a BigQuery client
client = bigquery.Client()

# Define the BigQuery dataset and table
dataset_id = 'your_dataset_id'
table_id = 'your_table_id'

# Load the DataFrame into BigQuery table (Step 2)
job_config = bigquery.LoadJobConfig()
job = client.load_table_from_dataframe(df, f"{dataset_id}.{table_id}", job_config=job_config)
job.result()  # Wait for the job to complete

The issue is specifically when the job config's source_format is not specified to CSV. I understand this means the source_format defaults to Parquet which does not support having spaces in column names. However, this is not obvious from the user's perspective so I am wondering if there should be some error if a column name has a space in it (or any other characters that are not supported).

[tswast]

Since spaces in column names are a new feature https://cloud.google.com/bigquery/docs/release-notes#March_22_2023 I'll mark this as a feature request.

Do we know if this is a limitation of the parquet format or the BigQuery parquet loader?

[chalmerlowe]

The issue with spaces in column names appears to be a well-known issue with the parquet format itself.
The standard solution shown on the Internet is to create column names that do not have spaces, i.e. if given: Full Name one might try:

• FullName
• Full_Name
• Fullname

The downside/risk of implementing an automagic column name reformatter is that it will mangle the user's column names:

• invisibly, without user knowledge leading to difficult to debug troubleshooting
• might lead to column name collisions (i.e. if the dataset already has two columns Product ID and ProductID, then how does the reformatter resolve this?)

I prefer to leave the user in charge of understanding their data and handling the names and naming of their data manually rather than seeking to invisibly manipulate it for them.

Closing as will not fix.