[Bug]: Poor performance of taps

[Jack-Burnett]

Singer SDK Version

0.13.1

Python Version

3.10

Bug scope

Taps (catalog, state, stream maps, etc.)

Operating System

MacOS

Description

I've been trying to optimise the performance of a tap i am working on, and find that a lot of it's time is spent in the code of the library itself.
Roughly 0.5-0.7 seconds per 1000 records.

Here is an excerpt from a profiled run (via pycharm);

As you can see, it spends around half of it's total time on the to_dict calls inside format_message.

If we switch line 165 in messages.py from;
return json.dumps(message.to_dict(), use_decimal=True, default=str)
to
return json.dumps(message.__dict__, use_decimal=True, default=str)
it basically entirely eliminates this and doubles the throughput of the tap.

My understanding is that dict just returns the existing object dict, whereas to_dict does a complex deep clone.

In my personal tests this has no downsides, all the tests still pass (didn't try the sdk tests yet), though I'm not sure if there are ramifications beyond that.

Thoughts?

Code

No response

[Jack-Burnett]

Made a PR with my proposed improvement here: #1196
My initial suggestion of using dict had some issues with BatchMessage, the solution in this PR seems pretty clean though.
The same change could (should?) be applied to the other message types too, though RecordMessage is of course by far the most common one.

[tayloramurphy]

@Jack-Burnett thanks for opening the issue and PR! I've added it to our Engineering Board to have one of the engineers take a look 😄 cc @edgarrmondragon @aaronsteers

[edgarrmondragon]

@Jack-Burnett thanks for reporting! You're absolutely right that asdict does a deepcopy:

Converts the dataclass obj to a dict (by using the factory function dict_factory). Each dataclass is converted to a dict of its fields, as name: value pairs. dataclasses, dicts, lists, and tuples are recursed into. Other objects are copied with copy.deepcopy().

https://docs.python.org/3/library/dataclasses.html

This was my oversight when re-implementing the message classes.

[Jack-Burnett]

That was a quick turnaround! Thanks