Why are array and empty object types flattened to string when flattening is enabled?

[ReubenFrankel]

sdk/singer_sdk/helpers/_flattening.py

Lines 333 to 339 in 3c076fc

	elif "array" in field_schema["type"] or (
	"object" in field_schema["type"] and max_level > 0
	):
	types = (
	["string", "null"] if "null" in field_schema["type"] else "string"
	)
	items.append((new_key, {"type": types}))

Specifically object type with no properties getting flattened is not desirable for us. Why is this the case?

[ReubenFrankel]

A little more context: basically we weren't seeing any data come through for a property of type object with no defined properties when applying flattening directly in mapper-fivetran. Turns out the additional schema transformations we were making meant that the schema used to lookup property keys during record flattening was no longer just the "flattened schema" (as the method argument implies). We solved this by preserving the flattened schema and overriding flatten_record to reference that specifically: Matatika/mapper-fivetran@37e2ebf

Maybe this is a sign we were/are not following best-practice when implementing our own transformation in a custom mapper, or perhaps exposes an enhancement that can be made to the SDK. Either way, it would still be good to know some background on why these types get flattened to string, given that they already exist as single properties?

[tayloramurphy]

Pinging @edgarrmondragon here for visibility

[edgarrmondragon]

Why are array and empty object types flattened to string when flattening is enabled?

The short answer is that flattening assumes that the intention is to fit the record into a format where a JSON object or array would be problematic, e.g. a CSV.

I'm happy to change that behavior if it turns out it's unexpected or undesirable for a majority of users.