Smoke tests: Fix random seed on smoke tests and add asserts on results

[lotif]

PR Type

Other

Short Description

Clickup Ticket(s): https://app.clickup.com/t/8686mur37

• Passing a random seed to FedProx, APFL and SCAFFOLD
• Making them save the metrics to a file at the end of their execution
• Adding assertions to their metrics values in the smoke tests

Tests Added

The smoke tests themselves.

[emersodb]

I may be missing something, so correct me if I'm wrong, but this looks like we're going to compare any metrics json dumped by a client to a single metrics_to_assert dictionary. For some of our example, that's definitely fine, because the different clients are loading the same dataset and running the same training cycle. However, there are instances where we might want to test when that isn't the case. That is, each client loads a distinct dataset. So their metrics won't be the same as each other.

Again, correct me if that interpretation is wrong.

[lotif]

That's correct, but I think the solution might be somewhere else. I think for that use case we could pass in different dictionaries for each client, giving each a client name or id, and then compare them appropriately when pulling their json files. Thoughts?

[emersodb]

Yes, I think that makes sense. We can do another smoke test where two clients load slightly different datasets and include it as a separate test.

[emersodb]

Overall it looks really great. Some fairly minor comments. The only one of moderate significance is the assumption that each client will produce matching metrics. In the general case, they won't. So we probably want that flexibility built into the smoke test infra.