Inconsistent display of indices for Data Editor with dynamic number of rows

[MathCatsAnd]

Checklist

• I have searched the existing issues for similar issues.
• I added a very descriptive title to this issue.
• I have provided sufficient information below to help reproduce this issue.

Summary

When a data editor is set to num_rows='dynamic' the index column is hidden when it is an "auto index" but displays otherwise. The hidden index is expected for an editor with dynamic rows, but there are inconsistencies as noted below.

The index can be forced to display through manually assigning values to the index or by using masking to generate a new dataframe. The use case in which I came across this was in performing validation operations on user input provided through a data editor; I would need to perform extra steps to make sure the index did not show up after the resultant dataframe was re-seeded as the beginning for a new data editor.

Reproducible Code Example

import streamlit as st
import pandas as pd

df = pd.DataFrame({'A':[1,2,3],'B':[1,2,3]})

cols = st.columns(3)
df = df.reset_index()
cols[0].write('Dataframe display')
cols[0].dataframe(df)
cols[1].write('Data Editor display')
cols[1].experimental_data_editor(df, num_rows='dynamic')
cols[2].write('Column Types')
cols[2].write(df.dtypes)

cols = st.columns(3)
df.index = ['a','b','c']
cols[0].write('Dataframe display')
cols[0].dataframe(df)
cols[1].write('Data Editor display')
cols[1].experimental_data_editor(df, num_rows='dynamic')
cols[2].write('Column Types')
cols[2].write(df.dtypes)

cols = st.columns(3)
df = df.reset_index()
cols[0].write('Dataframe display')
cols[0].dataframe(df)
cols[1].write('Data Editor display')
cols[1].experimental_data_editor(df, num_rows='dynamic')
cols[2].write('Column Types')
cols[2].write(df.dtypes)

Steps To Reproduce

Run the code and observe the following:

1. When the reset_index method is used, a new column named 'index' is created, but is not visible in the data editor.
2. When the index is manually assigned from a list, then it will show on the data editor (and the previously hidden column named 'index' shows up as well).
3. When the index is reset again, the previously assigned manual index displays in the data editor (column name 'level_0'), but the first reset index (column name 'index') and the current index both don't show.

Expected Behavior

As I understand it, the index is meant to be hidden on the dynamic row editor, at least when it is an auto index. I would expect at least columns created from an index via reset_index would be visible like normal columns.

With a new release set to give a kwarg for hiding the index or not, this may already be improved, but I'm noting the full extent of the inconsistency so it can be tracked. This may simply result in further documentation of "known limitations."

Current Behavior

Auto-indices do not show (expected).
Non-auto-indices do show (may or may not be expected?)
Columns derived from (previous) auto-indices do not show if there is currently an auto index (not expected)
Columns derived from (previous) auto-indices do show if there is not currently an auto index (expected)

Is this a regression?

• Yes, this used to work in a previous version.

Debug info

• Streamlit version: 1.22
• Python version: 3.10.11
• Operating System: Win 10
• Browser: Firefox
• Virtual environment: Conda

Additional Information

No response

Are you willing to submit a PR?

• Yes, I am willing to submit a PR!

[MathCatsAnd]

I was going to submit this as a separate issue, but I will note it here for now since it's related:

When an index is visible with num_rows='dynamic', the data editor will not publish a new row until an index is manually entered. The data for the row is shown in the session state value if a key is assigned, but without a value specified in the index, the row will not be rendered and not returned in the resultant dataframe.

[lukasmasuch]

@MathCatsAnd Thanks for reporting this issue. The upcoming release will make this a bit more consistent:

But there is one aspect that we won't fix/change: If you set num_rows='dynamic' and your dataframe has a RangeIndex, we will hide this index. The reason is that there are technical complexities related to adding rows to a range index since we would need to determine the next valid number to use in the range index in the frontend. Theoretically, it would be possible to do that, but because of a few technical reasons we decided against doing this.

With the upcoming version, you will be able to still show the index by setting hide_index=False. However, the values in the range index for new rows will just show None: