Allow computing emissions and dumping results periodically for other output modes

[miquelmarti]

Currently, in the CSV output mode and other output modes results and emissions are only computed and persisted at the end of the run. I would rather get the results dumped to disk in the same way that for the API output mode emissions are computed and partial results persisted periodically:

codecarbon/codecarbon/emissions_tracker.py

Lines 700 to 713 in 63c6a55

	if (
	self._cc_api__out is not None or self._cc_prometheus_out is not None
	) and self._api_call_interval != -1:
	if self._measure_occurrence >= self._api_call_interval:
	emissions = self._prepare_emissions_data(delta=True)
	logger.info(
	f"{emissions.emissions_rate * 1000:.6f} g.CO2eq/s mean an estimation of "
	+ f"{emissions.emissions_rate*3600*24*365:,} kg.CO2eq/year"
	)
	if self._cc_api__out:
	self._cc_api__out.out(emissions)
	if self._cc_prometheus_out:
	self._cc_prometheus_out.out(emissions)
	self._measure_occurrence = 0

This way:

• If a run crashes unexpectedly partial results are still saved
• I can visualize the outputs during training
• I can monitor emissions by reading the outputs and preempt runs in case of increased emissions, for example

I think the easiest way would be to allow configuring which output modes should appear in the if statements in the lines above, but ideally one could configure different rates for different outputs.

[benoit-cty]

Hello,

Yes it could be nice to write to a CSV the same data as the API.

We have a flush() method that is more a checkpoint: it does not reset the data, it only store them to avoid loosing them if there is a crash and to give a view when training for a very long time.

See #438 (comment)

You have a parameter on_csv_write :

• update : the existing run_id row (erasing former data)
• append : add a new row to CSV file (defaults)

So it seems we already have all the parts needed for what you ask. Feel free to propose a PR.