Hubs cloud instance down

[zach-vz]

Hi there — As of 4:35pm ET yesterday, My Hubs Cloud setup appeared to be migrated automatically to a new EC2 instance (t3.medium). I received a message that my current setup was shutting down and a new one was spinning up. Sure enough — a new EC2 instance did spin up. But once it did, we lost access to our instance altogether. I tried updating the stack with a new EC2 instance to see if I just needed to manually “reboot” effectively, but when I did, it rolled back on error “Failed to receive 1 resource signal(s) for current batch. Each resource signal timeout is counted as a FAILURE.” In AppASG. No changes were made to my instance yesterday on AWS actively by me (the only one with access to our AWS console). The last activity I took on AWS was nearly 24 hours earlier when I setup two additional Hubs Cloud instances (on other domains entirely, which you wouldn’t think would have any impact on the one I already had, nor did they, at least not immediately). The Hubs Cloud instance in question continued to run properly after that activity for the next 22 hours just fine until this automated EC2 instance update. Any thoughts on what the issue might be here?

[zach-vz]

Update: I've shut down the entire stack and used CloudFormation to spin up a new one. It is up now, but now we cannot see the Admin panel at all. It is giving us a bunch of 500 errors. Any thoughts on what might cause this with a completely new / untouched build?

[zach-vz]

I've figured it out. Appears to be some sort of state issue with admin accounts where all but mine were in a sort of partial admin state but not fully admin capable. I've reset those accounts to be admin and things are working as expected again.

[netpro2k]

If you find out any more information about what caused the initial issue, or these accounts to get in this state, please do share so we can look into it.

[zach-vz]

@netpro2k -- If you want to replicate, it seems that if you delete a Hubs Cloud stack and then recreate it with the same name / details, it somehow holds onto the account holders' details previously created in the "old" stack, but leaves the accounts in a sort of semi-admin state where a previous admin account holder is able to access the admin panel, but only some (react?) components of it. The accounts are effectively made non-admin in that they can't effect key settings, but the state doesn't seemingly fully reflect that change on the front-end so they are able to get to routes they shouldn't be able to get to.

[zach-vz]

Fwiw, I put this in the category of low priority bugs, as long as someone can find this thread to know what to do if it happens to them! (#2643 seems like a more existential issue 😬 )