Implement disk RAID check before upgrades on larger platforms

[cdot65]

Is your feature request related to a problem? Please describe.

When upgrading PAN-OS on large firewalls with RAID storage using the pan-os-upgrade utility, reboot timeouts can occur due to the extended duration of the reboot process, which can take up to an hour. This issue arises when the upgraded firewall undergoes a RAID check during the reboot, causing significant delays. Currently, the utility does not have a mechanism to detect and handle RAID checks before initiating the upgrade process, leading to potential timeouts and incomplete upgrades.

Describe the solution you'd like

Enhance the pan-os-upgrade utility to include disk and RAID checks before performing an upgrade on firewalls with RAID storage. The utility should:

1. Before initiating the upgrade, run the show system raid command on the firewall to retrieve the RAID status and configuration.
2. Parse the output of the command and check the counter since the last RAID check took place.
3. If the counter indicates that a RAID check is overdue or likely to occur during the upgrade process, display a warning message to the user and prompt them to either proceed with the upgrade or abort the process.
4. Implement a log parsing function to search through the firewall logs using grep or a similar method to determine if the upgraded firewall is currently undergoing a RAID check.
5. If a RAID check is detected during the upgrade process, remove the peer firewall (if applicable) from the upgrade "revisit" list to avoid initiating an upgrade while the RAID check is in progress.
6. If no RAID check is detected or if the RAID check completes successfully, proceed with the normal upgrade process.

Describe alternatives you've considered

An alternative approach could be to force the RAID check to occur before the upgrade by triggering it manually. However, this might not always be feasible or desirable, as it could lead to extended downtime and may not align with the planned maintenance window.

Additional context

Here are a few additional points to consider:

• The threshold for determining when a RAID check is overdue or likely to occur should be configurable based on the specific requirements and best practices of the organization.
• The utility should provide clear and informative messages about the RAID status and any potential delays or actions taken due to RAID checks.
• In case of a RAID check being detected during the upgrade, the utility should log the details and provide guidance on when to reschedule the upgrade for the affected firewall.
• Consider adding a flag or configuration option to allow users to skip the RAID check if they have already performed it manually or have alternative monitoring in place.
• Update the project's documentation to include information about this new feature, explaining how it handles RAID checks and the impact on the upgrade process.

By implementing this feature, the pan-os-upgrade utility will proactively detect and handle scenarios where RAID checks may interfere with the upgrade process on firewalls with RAID storage. This will help avoid reboot timeouts, ensure successful upgrades, and provide a smoother experience for users managing large-scale firewall upgrades.