RecordField default_to_avro produces non serializable output

[dada-engineer]

Describe the bug
When having a nested model the avro_schema function of the model fails if the nested attribute contains a non json serializable field e.g. datetime.

To Reproduce

import dataclasses
from dataclasses_avroschema import AvroModel
from datetime import datetime

@dataclasses.dataclass
class Metadata(AvroModel):
     event_time: datetime = dataclasses.field(default_factory=datetime.utcnow, default=dataclasses.MISSING)

@dataclasses.dataclass
class Record(AvroModel):
    metadata: Metadata = dataclasses.field(default_factory=lambda: Metadata(), default=dataclasses.MISSING)

Record()
# Out: Record(metadata=Metadata(event_time=datetime.datetime(2023, 8, 5, 7, 32, 27, 798392)))

Record.avro_schema()

# Fails with: TypeError: Object of type datetime is not JSON serializable

Expected behavior
Should produce the avor schema with default of datetime as string

As for my understanding it is due to RecordField.default_to_avro simply calling the to_dict function on the value provided.
As far as I can tell the default_to_avro is only called on rendering, so it should be fine to do json.loads(value.to_json()) here.

Edit: fixed suggested fix to use to_json()

[dada-engineer]

Maybe not so urgent as this one should think if a dynamic value such as the result of datetime.now (probably often a default_factory) is really what you want in the schema (changes every time it is generated)

[michal-rogala]

We are seeing the same issue with static Decimal fields. It stopped working on 0.45.0 -> 0.45.1 version change.

Our schema:

 minimum_step: Decimal = Field(
        default=Decimal("0.10"),
    )

The 'default' key of OrderedDict AVRO schema contains:
'minimum_step': Decimal('0.10'),

which is not JSON serializable

[dada-engineer]

@michal-rogala what would you expect to happen? Parse as string into avro as it is more reliable than a float or render a float from decimal? Is the decimal type rendering a number?

[michal-rogala]

String is the most reliable way - I believe this issue is only about schema definition, right? I tested your fix and it works in our case too, but I didn't check what was the decimal representation there.

[dada-engineer]

I am sorry, checked avro spec and this projects spec and both are rendering it to bytes, I have no issues using decimal defaults. can you comment an example that raises?

[michal-rogala]

Yeah, I believe this bug can occur when there is a several nested records and the lowest one has a default decimal value. I tried to follow an execution trace and while individual sub-records schema are generated properly - the "stiched" final one has this issue.

[michal-rogala]

@dabdada

ok, I did some testing and the problem happens when there is a default factory specified for a subrecord:

class Level1(AvroBaseModel):
    minimum_step: Decimal = Field(
        default=Decimal("0.10"),
    )

class Level2(AvroBaseModel):
    l1: Level1 = Field(default_factory=Level1)

print(Level2.avro_schema())

[dada-engineer]

K thank you. Can you check if this is fixed by my branch as well? I do believe so

[michal-rogala]

yes, it seems so :)