how to download debug_examples images from clearml on python API

[davidenitti]

Is there a way to download the images saved on clearml for a given task (debug_examples) from python?

[davidenitti]

any idea?

[erezalg]

Hi @davidenitti,

Sorry for the slow reply, this somehow slipped my radar.

Short answer is yes, you can with this code:
from clearml import Task
from clearml.backend_api.services import events

print(Task._send(Task._get_default_session(), events.DebugImagesRequest([{"task": "3aef190f16b843509056afdf86344077", "metric": "Images"}])))

You should obviously change Task ID, metric and series to your own's.
Note that in SDK 1.6.3 and up, there is a bug that might prevent you from using it. If you're using an older version no problem, if not then you might need to wait a few day until we release a new version :)

The long answer BTW is that we'll add a better interface to retrieve debug samples 😄

[davidenitti]

thanks!

[davidenitti]

it seems that I can see only the last iteration image, is it possible to see also the past iterations?

[erezalg]

@davidenitti,
you can add iters=X, where X is the iterations window (how many iterations from the last one you'll retrieve).

Let me know if that works

[davidenitti]

I get this error:

    raise SendError(res, error_msg)
clearml.backend_interface.session.SendError: Action failed <500/100: events.debug_images/v2.7 (General data error (RequestError(400, 'x_content_parse_exception', '[1:459] [terms] failed to parse field [size]')))> (metrics=[{'task': '601d1e6eb485444da1ca414487a7799a'}], iters=100000)

[davidenitti]

@erezalg do you know how to fix this?

[jkhenning]

@davidenitti what size of window did you use?

[dnitti-psee]

@jkhenning 1000x1000 or maybe a bit bigger, does it matter?
I have the error only if I use inter, e.g.
res = Task._send(Task._get_default_session(), events.DebugImagesRequest([{"task":task.id}],iters=1000))
gives the error but this works
res = Task._send(Task._get_default_session(), events.DebugImagesRequest([{"task":task.id}]))

[jkhenning]

This API does aggregations on the debug image events, and is thus limited by size.
You should use the events.GetTaskEventsRequest request. Currently there's no task object interface for that. You can use the private function Task._get_all_events() (although it's not guaranteed to remain there as it's not part of the official interface), or at least take a look at it to see how to call this request 🙂

[dnitti-psee]

it kind of works now with res = Task._send(Task._get_default_session(), events.DebugImagesRequest([{"task":task.id}],iters=1000)) but it seems I don't get all the images.
can you make en example with the GetTaskEventsRequest and/or _get_all_events?
thanks

[erezalg]

@davidenitti

can you try calling and letting me know if it works?
res = task._get_all_events(max_events=None, batch_size=10000, event_type="training_debug_image")

[pollfly]

Hey @davidenitti! v1.10.0 is now out, supporting debug sample retrieval using Task.get_debug_samples()

[jkhenning]

@davidenitti closing this, please reopen if required.