Periodically call arbitrary callback

[jbweston]

Currently adaptive-scheduler is capable of periodically saving the learner data, and calling 'learner.to_dataframe' and saving the resultant dataframe.

I would like to request the option to be able to additionally call an arbitrary callback.
My use-case is that I would like to save the learner data in a more efficient format.

Failing this I could potentially override the 'save' method for my learners; does adaptive-scheduler assume anything about the filenames/data format produced by 'save'?

[basnijholt]

[This should be pretty easy to implement, because adaptive already accepts callbacks. We just need to pass the method and call it somewhere around here

adaptive-scheduler/adaptive_scheduler/_server_support/launcher.py

Line 145 in fee0703

runner.start_periodic_saving({"fname": fname}, interval=args.save_interval)

Currently AFK, but will look tomorrow.

[jbweston]

Yes; I think the issue is piping 'save_method' all the way from the top of the API. We would also need to store 'save_method' somewhere so that the clients can pick it up.

Contrast this to 'save_dataframe', where we hardcode the method invocation and instead pass through simple data (boolean flag and a format specifier string).

[basnijholt]

We can probably pass it in the same way as the runner_kwargs:

adaptive-scheduler/adaptive_scheduler/_server_support/launcher.py

Line 105 in fee0703

parser.add_argument("--serialized-runner-kwargs", action="store", type=str)

adaptive-scheduler/adaptive_scheduler/_server_support/job_manager.py

Line 73 in fee0703

base64_runner_kwargs = _serialize_to_b64(runner_kwargs)

[basnijholt]

@jbweston, I opened #209.