Skip to content

Spotfire Python — Pandas 3.0 Migration Guide

Jump to bottom
vrane-tibco edited this page Sep 23, 2026 · 2 revisions

Spotfire Python — Pandas 3.0 Migration Guide

We've officially upgraded spotfire-python to pandas 3.0! This release brings some significant improvements—like enforced Copy-on-Write (CoW) semantics, a dedicated string dtype, microsecond datetime resolution, and the cleanup of several outdated APIs. These changes are all about making your data workflows more efficient and accurate. However, because pandas 3 handles things a bit differently, you'll likely need to tweak a few of your existing scripts.

This guide is designed to walk you through what's changed in the spotfire-python API, the updates you'll need to make in your customer data function scripts, and how to use temporary opt-outs if you need a little more time to migrate.

Version Requirements

Dependency Previous New
Python >= 3.9 >= 3.11
pandas >= 2.2.2, < 3.0.0 >= 3.0.0

Let's talk about Metadata Access

With pandas 3.0, we're changing how we manage Spotfire metadata on DataFrames. Because of the new Copy-on-Write behavior, trying to access attributes directly on your DataFrames isn't just unreliable—it's going to trigger a warning. Even worse, if you try to set per-column metadata the old way (via df['col'].spotfire_column_metadata = ...), pandas will silently ignore it. Here is how you can adjust your scripts to keep things running smoothly.

Table Metadata

# OLD — still works but raises UserWarning in pandas 3
table_md = df.spotfire_table_metadata
df.spotfire_table_metadata = {'key': ['value']}

# NEW (recommended)
import spotfire
table_md = spotfire.get_table_metadata(df)
spotfire.set_table_metadata(df, {'key': ['value']})

Tip

While direct attribute access (like df.spotfire_table_metadata) still technically works for reading and writing, it's going to add warnings in logs every time you write to it. Plus, future versions of pandas might remove this entirely. To future-proof your code, we recommend switching to the new spotfire.get_table_metadata() and spotfire.set_table_metadata() functions. They use df.attrs under the hood, so they're clean, reliable, and warning-free.

Column Metadata

# OLD — silently lost under Copy-on-Write in pandas 3
col_md = df['col'].spotfire_column_metadata
df['col'].spotfire_column_metadata = {'key': ['value']}

# NEW (required — the old pattern is broken)
import spotfire
col_md = spotfire.get_column_metadata(df, 'col')
spotfire.set_column_metadata(df, 'col', {'key': ['value']})

Important

Unlike table metadata, the old way of modifying column metadata (df['col'].spotfire_column_metadata = ...) is completely broken in pandas 3. Because df['col'] now returns a copy, your metadata is being written to a temporary object that gets tossed out immediately. There's no workaround for this, so you'll need to migrate to the new API to keep your column metadata intact.

Spotfire Types

There are no changes to worry about. Both spotfire.get_spotfire_types() and spotfire.set_spotfire_types() continue to work exactly as they did before.

Copy Metadata

No breaking changes here either. spotfire.copy_metadata(source, destination) is still the way to go and works perfectly.

New Public API Functions

We've introduced these new tools to make managing your metadata much easier. Here is a quick reference for what they do:

Function Description
spotfire.get_table_metadata(df) Returns the table metadata dict
spotfire.set_table_metadata(df, metadata) Sets the table metadata dict
spotfire.get_column_metadata(df, col) Returns the column metadata dict for a column
spotfire.set_column_metadata(df, col, metadata) Sets the column metadata dict for a column

Existing Public API Functions

Function Description
spotfire.copy_metadata(source, dest) Copies all metadata between DataFrames (unchanged)
spotfire.get_spotfire_types(df) Returns Spotfire type names as a Series (unchanged)
spotfire.set_spotfire_types(df, types) Sets Spotfire type overrides for export (unchanged)

Pandas 3.0 Breaking Changes in Customer Scripts

The following changes are the ones most likely to affect your data function scripts. We've laid out the "broken" patterns and their corresponding fixes.

This guide covers the changes most likely to affect Spotfire data function scripts. For the complete list of pandas 3.0 changes, see the official pandas release notes: https://pandas.pydata.org/docs/whatsnew/v3.0.0.html

Copy-on-Write (Silent — No Error, Wrong Results)

Pandas 3.0 enforces Copy-on-Write. Chained assignment won't throw an error, but it silently fails to update your data:

# BROKEN — silently does nothing
df['price'][0] = 99.99
df['status'][df['status'] == 'pending'] = 'processed'

# FIX — use .loc
df.loc[0, 'price'] = 99.99
df.loc[df['status'] == 'pending', 'status'] = 'processed'
# BROKEN — filtered DataFrame is a copy, assignment lost
filtered = df[df['region'] == 'EMEA']
filtered['adjusted'] = filtered['value'] * 1.1

# FIX
df.loc[df['region'] == 'EMEA', 'adjusted'] = df.loc[df['region'] == 'EMEA', 'value'] * 1.1
# BROKEN — inplace on column slice
df['col'].fillna(0, inplace=True)

# FIX
df['col'] = df['col'].fillna(0)

inplace=True Methods Now Return Self Instead of None

Methods like fillna(), replace(), ffill(), bfill(), interpolate(), where(), mask(), and clip() used to return None when used with inplace=True. Now, they return the modified DataFrame. If your code checks for None, it's going to hit the wrong logic branch:

# BROKEN — result is now the DataFrame, not None
result = df.fillna(0, inplace=True)
if result is None:
    print("modified in place") # OLD: entered here
else:
    df = result # OLD: never reached; NOW: reached

# FIX — don't check return value, or stop using inplace
df.fillna(0, inplace=True)
# OR (recommended)
df = df.fillna(0)
# BROKEN — truthiness check
if df.replace({-1: np.nan}, inplace=True):
    # OLD: never entered (None is falsy)
    # NOW: always enters (DataFrame is truthy if non-empty)
    do_something()

# FIX — remove the conditional
df.replace({-1: np.nan}, inplace=True)

String Dtype Change

String columns now default to the str dtype rather than object:

# BROKEN — misses string columns in pandas 3
if df['name'].dtype == object:
    ...

# FIX
if pd.api.types.is_string_dtype(df['name']):
    ...

Need a quick fix? You can add this line at the top of your script to temporarily revert string columns to the old object dtype while you migrate (may not be available on all Python versions (e.g., removed on Python 3.14)):

pd.options.mode.infer_string = False

Frequency Aliases Removed

Old frequency aliases are no longer supported and will raise a ValueError:

# BROKEN
df.resample('M').mean()
df.resample('Q').sum()
pd.date_range('2024-01-01', periods=12, freq='Y')

# FIX
df.resample('ME').mean()
df.resample('QE').sum()
pd.date_range('2024-01-01', periods=12, freq='YE')
Old New Offset
M ME Month End
Q QE Quarter End
Y YE Year End
BM BME Business Month End
SM SME Semi-Month End
BQ BQE Business Quarter End
BY BYE Business Year End

Timezone Changes

Timezone objects have switched from pytz to zoneinfo.ZoneInfo:

# BROKEN
import pytz
tz = pytz.timezone('US/Eastern')
ts = tz.localize(datetime.datetime(2024, 1, 1))

# FIX
from zoneinfo import ZoneInfo
ts = pd.Timestamp(2024, 1, 1, tz='US/Eastern')
# BROKEN — pytz exception types
try:
    ts = pd.Timestamp('2024-11-03 01:30', tz='US/Eastern')
except pytz.exceptions.AmbiguousTimeError:
    pass

# FIX — catch ValueError instead
try:
    ts = pd.Timestamp('2024-11-03 01:30', tz='US/Eastern')
except ValueError:
    pass

DateTime Resolution

Default resolution has shifted from nanoseconds to microseconds:

# BROKEN — integer value is 1000x smaller than before
epoch_ns = df['timestamp'].astype('int64')

# FIX — explicitly specify resolution
epoch_ns = df['timestamp'].dt.as_unit('ns').astype('int64')

String Values Reject Non-String Assignment

String columns are stricter now and will reject non-string values:

# BROKEN — str dtype rejects non-strings
df = pd.DataFrame({'col': ['a', 'b', 'c']}) # dtype: str in pandas 3
df.loc[0, 'col'] = 123 # TypeError

# FIX — cast the value or convert column dtype
df.loc[0, 'col'] = str(123)
# OR
df['col'] = df['col'].astype(object)
df.loc[0, 'col'] = 123

Null Checking with Strings

String columns now use NaN as the missing sentinel instead of None:

# BROKEN — None check fails for string columns
for val in ser:
    if val is None: # misses NaN values in pandas 3
        print("missing")

# FIX — use pd.isna() which handles both
for val in ser:
    if pd.isna(val):
        print("missing")

Performance Impact

We've tested SBDF export performance against spotfire-python 2.4.2 (pandas 2.3.3) to make sure there are no regressions. The new version utilizes an optimized internal Copy-on-Write dict for metadata and uses fillna instead of replace for NA handling, so you should see equal or better performance.

SBDF Export Benchmarks (3 rows, varying column count, with column metadata)

Tested with original spotfire-python 2.4.2 + pandas 2.3.3 vs new spotfire-python + pandas 3.0.3:

Columns spotfire 2.4.2 (pandas 2) New (pandas 3) Ratio
100 0.019s 0.016s 0.8x
500 0.084s 0.057s 0.7x
1,000 0.189s 0.118s 0.6x
2,000 0.427s 0.339s 0.8x
5,000 1.327s 1.289s 1.0x
10,000 4.417s 4.402s 1.0x

No performance regression. In fact, the new version matches or outperforms the original at all column counts, and your file sizes will remain identical.

Benchmark Script

You can use this script to test the SBDF export performance in your own environment:

import time, os, tempfile, warnings
import pandas as pd
from spotfire import sbdf

try:
    from spotfire import _metadata
    USE_NEW_API = True
except ImportError:
    USE_NEW_API = False

for n in [100, 500, 1000, 2000, 5000, 10000]:
    df = pd.DataFrame({f"num{i}": [i, i+1, i+2] for i in range(n)})

    # Set column metadata using the appropriate API
    if USE_NEW_API:
        for i in range(n):
            _metadata.set_column_metadata(df, f"num{i}", {f"k{i}": [i]})
    else:
        with warnings.catch_warnings():
            warnings.simplefilter("ignore")
            for i in range(n):
                df[f"num{i}"].spotfire_column_metadata = {f"k{i}": [i]}

    f = tempfile.NamedTemporaryFile(suffix=".sbdf", delete=False)
    f.close()
    start = time.time()
    sbdf.export_data(df, f.name, "test")
    elapsed = time.time() - start
    os.unlink(f.name)
    print(f"columns={n:6d} export: {elapsed:.3f}s")

Quick Migration Checklist

  • Replace df.spotfire_table_metadata with spotfire.get_table_metadata(df) / spotfire.set_table_metadata(df, ...)
  • Replace df['col'].spotfire_column_metadata with spotfire.get_column_metadata(df, 'col') / spotfire.set_column_metadata(df, 'col', ...)
  • Replace chained assignment (df['col'][idx] = val) with .loc (df.loc[idx, 'col'] = val)
  • Replace df['col'].fillna(0, inplace=True) with df['col'] = df['col'].fillna(0)
  • Remove any checks for inplace=True returning None — it now returns the DataFrame
  • Replace dtype == object checks with pd.api.types.is_string_dtype()
  • Replace select_dtypes(include=['object']) with select_dtypes(include=['object', 'string'])
  • Replace frequency aliases: M → ME, Q → QE, Y → YE, BM → BME, BQ → BQE, BY → BYE
  • Replace pytz usage with zoneinfo
  • Replace val is None null checks with pd.isna(val) for string columns
  • Update Python to 3.11+
  • Add spotfire.copy_metadata(source, result) after merge() operations if metadata preservation is needed

Temporary Opt-Outs

If you can't update everything all at once, these options can help you migrate gradually:

Opt-out What it does Limitation
pd.options.mode.infer_string = False Reverts string columns to object dtype Does not affect explicitly typed string columns
Copy-on-Write No opt-out available — CoW is always on in pandas 3 Must update chained assignment patterns

Note

pd.options.mode.infer_string may not be available on all Python versions (e.g., removed on Python 3.14). Check availability before relying on this opt-out. Plan to remove it before pandas 4, where the opt-out will be removed.

Clone this wiki locally