XCS/Monsoon2 currently unavailable (Monday 27th January)

[Roger Milton (mo-roger-milton)]

Dear all,

following power supply issues in the area at approximately 18:30 yesterday (Sunday 26th January 2025) the XCS supercomputer which hosts the Monsoon2 service suffered a failure requiring it to be rebooted. The system has now been recovered, but workload queues have not been re-enabled. We intend to re-enable work queues once we are confident that the power supply is stable again.

Please accept my apologies for yet another interruption to service, and we will update you again when we know we will be re-enabling the queues.

[Roger Milton (mo-roger-milton)]

Please see this update on re-enabling the queues, in summary, this isn't expected to be before 6am tomorrow (28th January):

Just as an update of the current XCS situation:

We have confirmed, via the IMT (Met Office Incident Management Team) chair, that until the IT Hall 3 UPS batteries are replaced, all running work on XCS could be stopped based on potential electrical disruption from severe weather events.

This will not prevent the system failing if there is an interruption to power, but we believe it lessens the risk to the file system for significant damage to disks and data loss or corruption.

The current weather warning for strong winds expires at 6am tomorrow morning (Tuesday 28th January).

Current estimate for IT Hall 3 battery replacement is mid-February.

I appreciate this will be quite an inconvenience for Research and Collaboration users, but the decision is driven by the desire to limit any potential damage to our aged Supercomputer systems, in hopefully the last 6 months of their lives.

[mo-mahammedk]

Update: Queues have been re-enabled, and are starting to assign work. We shall be monitoring the system and will update you if anything changes.