Skip to content

Ensure training memory monitor cleans up callbacks#96

Merged
lukifer23 merged 1 commit intomasterfrom
codex/add-memory-monitoring-with-unregistration-support
Sep 23, 2025
Merged

Ensure training memory monitor cleans up callbacks#96
lukifer23 merged 1 commit intomasterfrom
codex/add-memory-monitoring-with-unregistration-support

Conversation

@lukifer23
Copy link
Owner

Summary

  • extend the memory monitor helpers to report start/stop success and expose a callback removal API
  • teach train_comprehensive to track monitor lifecycle, unregister its alert callback, and stop monitoring during cleanup
  • add a regression test that runs training twice to verify callbacks are not duplicated and the monitor thread stops

Testing

  • pytest tests/test_train_memory_monitor.py

https://chatgpt.com/codex/tasks/task_e_68d1c2a371b88323861a41c2fc7d9bc8

Copilot AI review requested due to automatic review settings September 23, 2025 13:38
@lukifer23 lukifer23 merged commit 6211c78 into master Sep 23, 2025
1 check failed
@lukifer23 lukifer23 deleted the codex/add-memory-monitoring-with-unregistration-support branch September 23, 2025 13:38
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR implements proper cleanup for the memory monitoring system to prevent resource leaks and callback accumulation. The training function now tracks the memory monitor's lifecycle and ensures proper shutdown when training completes or fails.

  • Extended memory monitor API to report start/stop success and expose callback removal functionality
  • Modified train_comprehensive to properly manage monitor lifecycle and clean up registered callbacks
  • Added comprehensive regression test to verify no callback duplication and proper thread cleanup

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File Description
tests/test_train_memory_monitor.py New test file with regression test for memory monitor cleanup
azchess/utils/memory_monitor.py Added return values to start/stop methods and callback removal API
azchess/utils/init.py Exported new remove_memory_alert_callback function
azchess/training/train.py Added monitor lifecycle tracking and cleanup in train_comprehensive

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment on lines +49 to +50
while False:
yield None
Copy link

Copilot AI Sep 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This generator will never yield any values since the while condition is False. This should be while True: or use a different approach to create an empty generator.

Suggested change
while False:
yield None
if False:
yield

Copilot uses AI. Check for mistakes.
"""Stop the memory monitoring thread."""
if not self.is_monitoring:
return
if not self.is_monitoring and not (self.monitor_thread and self.monitor_thread.is_alive()):
Copy link

Copilot AI Sep 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nitpick] This condition check is confusing. Consider simplifying to check if there's nothing to stop: if not self.is_monitoring and (not self.monitor_thread or not self.monitor_thread.is_alive()):

Suggested change
if not self.is_monitoring and not (self.monitor_thread and self.monitor_thread.is_alive()):
if not self.is_monitoring and (not self.monitor_thread or not self.monitor_thread.is_alive()):

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants