add unit tests for convert_to_model_format and fix numpy deprecation (Creating an ndarray from ragged nested sequences)

[ka-bu]

What problem are you trying to solve?

Numpy >= 1.19 will complain about creating arrays from lists with different shape, e.g.

np.array([ np.array([1,3]), np.array([1]) ] ) :1: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray array([array([1, 3]), array([1])], dtype=object)

This happens whenever we collect our Features in a FeatureArray (but not inside of FeatureArray but when we initialize the FeatureArray with an np.array( [ .... features ] )).

What's your suggested solution?

in model_data_utils.py and model_data.py:

• check cases in which irregular shapes are combined (should be the case for all "sequence" features and all features computed for dialogues because of varying dialogue lengths, plus some masks)
• set dtype=object when combining irregular shaped arrays into one array (dtype=None otherwise)
• adapt checks of FeatureArray that'll fail because of the changed dtypes

Examples (if relevant)

No response

Is anything blocking this from being implemented? (if relevant)

No response

Definition of Done

• added unit tests for convert_to_data_format (which uses _feature_arrays_for_attribute which contains some of the FeatureArray instantiations causing the warning)
• removed the filtering of the Depcrecation warning from common.py
• new and existing unit tests pass without raising the numpy deprecation warning

[sync-by-unito]

➤ Maxime Verger commented:

💡 Heads up! We're moving issues to Jira: https://rasa-open-source.atlassian.net/browse/OSS.

From now on, this Jira board is the place where you can browse (without an account) and create issues (you'll need a free Jira account for that). This GitHub issue has already been migrated to Jira and will be closed on January 9th, 2023. Do not forget to subscribe to the corresponding Jira issue!

➡️ More information in the forum: https://forum.rasa.com/t/migration-of-rasa-oss-issues-to-jira/56569.