GE errors when trying to return a result when contains a pandas NA

[isichei]

Describe the bug

If asking GE for a complete result when expect_column_values_to_not_be_null. GE will try to return a result which contains samples of the values which failed the expectation (if result is set to COMPLETE). It fails when one of the values is the new pandas NA type (GE internals fails to convert it).

To Reproduce

import great_expectations as ge
import pandas as pd

df = pd.DataFrame({"a": [1, pd.NA, 3]})
dfe = ge.dataset.PandasDataset(df)
dfe.expect_column_values_to_not_be_null("a", result_format="COMPLETE") # Will error

Error message:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "env/lib/python3.8/site-packages/great_expectations/core/__init__.py", line 2000, in __repr__
    return json.dumps(self.to_json_dict(), indent=2)
  File "env/lib/python3.8/site-packages/great_expectations/core/__init__.py", line 2026, in to_json_dict
    myself = expectationValidationResultSchema.dump(self)
  File "env/lib/python3.8/site-packages/great_expectations/marshmallow__shade/schema.py", line 556, in dump
    processed_obj = self._invoke_dump_processors(
  File "env/lib/python3.8/site-packages/great_expectations/marshmallow__shade/schema.py", line 1075, in _invoke_dump_processors
    data = self._invoke_processors(
  File "env/lib/python3.8/site-packages/great_expectations/marshmallow__shade/schema.py", line 1234, in _invoke_processors
    data = processor(data, many=many, **kwargs)
  File "env/lib/python3.8/site-packages/great_expectations/core/__init__.py", line 2096, in convert_result_to_serializable
    data.result = convert_to_json_serializable(data.result)
  File "env/lib/python3.8/site-packages/great_expectations/core/__init__.py", line 132, in convert_to_json_serializable
    new_dict[str(key)] = convert_to_json_serializable(data[key])
  File "env/lib/python3.8/site-packages/great_expectations/core/__init__.py", line 139, in convert_to_json_serializable
    new_list.append(convert_to_json_serializable(val))
  File "env/lib/python3.8/site-packages/great_expectations/core/__init__.py", line 195, in convert_to_json_serializable
    raise TypeError(
TypeError: <NA> is of type NAType which cannot be serialized.

Expected behavior
To return a result and not fail. Maybe with pd.NA or just replace it with a string interpretation of pd.NA or a None as the returned list of values (which failed).

Environment (please complete the following information):

• OS: [Mac]
• GE Version: 0.12.6
• Pandas Version: 1.1.3

[eugmandel]

@isichei Thank you for reporting this! If you would like to make a PR (the fix must be very small), it would be super welcome! Please let us know.

[isichei]

Yeah can try and have a look over the next couple of weeks. When you say the fix must be very small do you mean it will be a small fix or just that you try to keep PRs small?

If it is a bigger fix would you want me to make several small PRs?

[eugmandel]

@isichei I meant "must" not as in "we require the fix to be small", but as in "I think it will not take much code to fix this" :)

Looking forward to your PR!