DEVOPS-1973 improve stack health reporting

[kkellerlbl]

The Rancher 1 API doesn't report stack health very granularly. There are many legitimate reasons the stack could be degraded but still fine. (For example, a new service is being spun up right when the health check is done.)

These changes try to track specifically which services (if any) are not healthy when the stack reports unhealthy. If all services are now healthy when checked, assume the stack is too and don't throw an alarm. And if a service is unhealthy, track it in a sqlite database, and only report the stack unhealthy if there are services that have been unhealthy for longer than the stack_health_age parameter. This avoids the issue of two otherwise healthy services causing the entire stack to seem unhealthy for longer than that parameter.

This PR also removes some old code.

[bio-boris]

Pull Request Review

General Feedback

This script checks the status of Rancher 1.x agents, stacks, and services in specified environments and creates a dummy service in a given stack if required. The code is generally well-structured but can benefit from improvements in error handling, modularity, and readability.

Specific Suggestions

1. Imports and Dependencies

• Issue: Multiple imports are unused
• Suggestion: Remove unused imports to improve code clarity.

2. Argument Parsing

• Issue: Argument parsing and configuration file loading are mixed with the main logic.
• Suggestion: Encapsulate argument parsing and configuration loading in separate functions to enhance modularity.

3. Error Handling

• Issue: Error handling is minimal, especially in HTTP requests and JSON parsing.
• Suggestion: Add robust error handling to manage potential failures in network requests and JSON parsing.

4. Code Modularity

• Issue: The process_section function is lengthy and handles multiple responsibilities.
• Suggestion: Break down process_section into smaller, single-responsibility functions (e.g., check_rancher_agents, monitor_services, test_stack_health, create_dummy_service).

5. Logging

• Issue: The script prints directly to the console.
• Suggestion: Use a logging framework to provide better control over logging levels and outputs.

6. Configuration Management

• Issue: Configuration parameters are scattered and not validated.
• Suggestion: Centralize configuration management and validate parameters before use.

7. Code Style

• Issue: Inconsistent code style (e.g., mix of single and double quotes, inconsistent indentation).
• Suggestion: Follow PEP 8 style guidelines to ensure consistent and readable code.