[Bug]: flattening_enabled turns arrays in record to strings without changing the schema type

[Jake-Gillberg]

Singer SDK Version

0.14.0

Python Version

3.10

Bug scope

Taps (catalog, state, stream maps, etc.)

Operating System

Linux

Description

When setting "flattening_enabled" : true, record property values of array ([]), are turned to strings ("[]"), but the schema still lists the property as "type" : ["null", "array"], causing an error when this hits a meltano sdk target of:

jsonschema.exceptions.ValidationError: '[]' is not of type 'null', 'array'

Failed validating 'type' in schema['properties']['<property name>']:
{'items': {}, 'type': ['null', 'array']}
On instance['<property  name>']:
'[]'

Code

No response

[tayloramurphy]

@edgarrmondragon can you confirm this is still a bug?

[kgpayne]

I hit this today whilst working on making target tests pass and didn't have time to investigate further. Couldn't find in the docs what the expected behaviour is, but we do have a separate config bool called flatten_arrays that I think defaults to True that might have something to do with it. In any case, the mismatch to the schema points to a bug 🐛

[edgarrmondragon]

@edgarrmondragon can you confirm this is still a bug?

@tayloramurphy Yes, and it most likely has the same underlying cause as #1400

[edgarrmondragon]

This was fixed in #1939. See the "some_numbers" test case in the PR for the expected output.