[Feature Request] Upload Artifact Fails on Non-Standard Dictionaries

[jalexand3r]

Currently upload_artifact only checks whether an object is a dict before dumping to JSON:

clearml/clearml/binding/artifacts.py

Line 380 in 51d70ef

elif isinstance(artifact_object, dict):

However, there is no guarantee that values within the dictionary are JSON serializable (e.g. dictionaries of DataFrames will fail).

Although it may not be best practice to upload dictionaries of objects instead of uploading each object individually, there are good use cases for the former. It would be nice to have error handling that defaults to pickling objects if auto_pickle is true and standard data type uploading fails.

[bmartinn]

Hi @jalexand3r

Although it may not be best practice to upload dictionaries of objects instead of uploading each object individually, there are good use cases for the former. It would be nice to have error handling that defaults to pickling objects if auto_pickle is true and standard data type uploading fails.

This is a good idea. So let's assume we try to upload a non-jsonable dict, we output a warning (e.g JSON serialization failed, falling back to pickle), then we would just pickle the dict itself, and upload the pkl file as artifact.
Is this what you had in mind?

[jalexand3r]

I would check the auto_pickle argument after the warning and before pickling, but yes, that is essentially what I had in mind.

[bmartinn]

Sounds good to be, @jalexand3r do you mind renaming the issue to feature request ?

[jkhenning]

Closing this as it was already released. Please reopen if required.