adding hardware usage and software packages tracker

[abidwael]

This provides a new Tracker class to track hardware usage and software packages while a block of code is being executed.

Usage:

with Tracker(tag='train', output_dir=model.config['backend']['cache_dir'], num_batches=model.config[TRAINER]["batch_size"], num_examples=len(training_set)) as tracker:
    # code block
    .
    .

Will save hardware and software usage metrics under f"{output_dir}/{tag}_metrics.json"

[github-actions]

Unit Test Results

       6 files  +    1         6 suites  +1   2h 37m 9s ⏱️ + 36m 43s
2 913 tests  -   16  2 868 ✔️  -   15    45 💤  -   1  0 ❌ ±0 
8 739 runs  +123  8 600 ✔️ +103  139 💤 +20  0 ❌ ±0 

Results for commit c18b6bd. ± Comparison against base commit 9c58d5e.

♻️ This comment has been updated with latest results.

[ShreyaR]

Thanks for putting up this PR! Left some comments.

One general question I have is what the overhead of using Tracker is. If it isn't too expensive or slow to run Tracker, then it may make sense to use it by default for any Ludwig training process. I can see it being quite useful if we have a json with all the benchmarking stats generated in the model artifacts folder anytime we do training/evaluation.

[abidwael]

Should this be a separate requirements_tracker.txt file or do I need to add it to the main requirements.txt file?

[justinxzhao]

I'd be ok with adding this to the main requirements.txt file, especially if hardware resource usage tracking adds marginal overhead.

Curious about other people's opinions on this: @dantreiman @w4nderlust @tgaddair

[abidwael]

Thanks for putting up this PR! Left some comments.

One general question I have is what the overhead of using Tracker is. If it isn't too expensive or slow to run Tracker, then it may make sense to use it by default for any Ludwig training process. I can see it being quite useful if we have a json with all the benchmarking stats generated in the model artifacts folder anytime we do training/evaluation.

@ShreyaR I ran model.experiment with and without Tracker for ames_housing and mercedes_benz_greener and collected the total cpu and ram usage for all processes running on the machine at the time of execution. Here are the results:

CPU seems to be more or less unaffected, but there's some RAM overhead. In my opinion, it's worth adding an optional Tracker ctx per @justinxzhao 's suggestion.

[justinxzhao]

I'd be ok with adding this to the main requirements.txt file, especially if hardware resource usage tracking adds marginal overhead.

Curious about other people's opinions on this: @dantreiman @w4nderlust @tgaddair

[justinxzhao]

Discussed offline: Add a basic unit test that shows how this class can/should be used.

[abidwael]

The pre-commit.ci check will not pass because of the following block

# disabling print because the following imports are verbose
f = open(os.devnull, "w")
sys.stdout = f
from experiment_impact_tracker.cpu.common import get_my_cpu_info
from experiment_impact_tracker.gpu.nvidia import get_gpu_info
from experiment_impact_tracker.py_environment.common import get_python_packages_and_versions

f.close()
sys.stdout = sys.__stdout__

I'm temporarily redirecting stdout because the import statement is verbose.

Made a PR in the original repo: Breakend/experiment-impact-tracker#74
Will follow up with the maintainer.

[justinxzhao]

Changes LGTM, looks like there's a few last errors to resolve:

From pre-commit:

ludwig/benchmarking/resource_usage_tracker.py:24: [E402] module level import not at top of file
ludwig/benchmarking/resource_usage_tracker.py:25: [E402] module level import not at top of file
ludwig/benchmarking/resource_usage_tracker.py:26: [E402] module level import not at top of file

Finally, could you check the unit test you added? It looks like it's failing on one of the builds.

[abidwael]

Changes LGTM, looks like there's a few last errors to resolve:

From pre-commit:

ludwig/benchmarking/resource_usage_tracker.py:24: [E402] module level import not at top of file
ludwig/benchmarking/resource_usage_tracker.py:25: [E402] module level import not at top of file
ludwig/benchmarking/resource_usage_tracker.py:26: [E402] module level import not at top of file

Finally, could you check the unit test you added? It looks like it's failing on one of the builds.

This one is due to my previous comment here.

[justinxzhao]

LGTM after the pre-commit error is fixed, and the tests are all green.