QST: With to_dict method, whether support batch transfer pd.NA in dataframe to None in dict value

[flyly0755]

Research

• I have searched the [pandas] tag on StackOverflow for similar questions.

• I have asked my usage related question on StackOverflow.

Link to question on StackOverflow

https://stackoverflow.com/questions/43254699/pandas-to-dict-returns-none-mixed-with-nan

Question about pandas

pddata as below:
intc strc floatc
0 1 a 1
1 <NA> b 2
2 3 c 3

listdata = pddata.to_dict('records')

then listdata with value pd.NA, which is not convenient to check in the code,
[{'intc': 1, 'strc': 'a', 'floatc': 1}, {'intc': , 'strc': 'b', 'floatc': 2}, {'intc': 3, 'strc': 'c', 'floatc': 3}]
usually need loop to change all the pd.NA to None.
Whether pddata.to_dict support this data transfer and
with better performence compare to python list loop method when process large of data,
for example, a list of 10million dict element.
If not support, maybe in the futher enhence to_dict method to support it?

[topper-123]

Can you give a copy-pasteable code example that demonstrates the problem, please, so I can take a look at this.

[flyly0755]

dftest = pd.DataFrame({
    'intc': [1, pd.NA, 3],
    'strc': ['a', 'b', 'c'],
    'floatc': [1.0, 2.0, 3.0],
    'intc2': [1, pd.NA, pd.NA],
})

with pandas data as above, i want to transform to list with dict element
listdata = pddata.to_dict('records')
but after that, i need to loop listdata for changing every pd.NA to general None
This is the situation with pandas 1.5.

But with pandas 2.0, the result is different, default change pd.NA to None
[{'intc': 1, 'strc': 'a', 'floatc': 1.0, 'intc2': 1},
{'intc': None, 'strc': 'b', 'floatc': 2.0, 'intc2': None},
{'intc': 3, 'strc': 'c', 'floatc': 3.0, 'intc2': None}]

[topper-123]

Do you think the old or new behaviour is better? I got trouble understanding what your question, sorry.

[flyly0755]

The new behaviour is what i want, thx! so changing pandas version from 1.5 to 2.0 can solve this problem.