Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resolver Cloud Issues #600

Closed
Toflar opened this issue Oct 12, 2020 · 3 comments
Closed

Resolver Cloud Issues #600

Toflar opened this issue Oct 12, 2020 · 3 comments
Assignees

Comments

@Toflar
Copy link
Member

Toflar commented Oct 12, 2020

There have been stability issues with the https://composer-resolver.cloud over the last two weeks which were all caused by multiple different problems.

First of all, we've released Contao Manager 1.3.0 which contained an issue that caused too many requests to the Cloud Resolver. Combined with the fact that we've released security releases for Contao, the demand was so high that it reached around 250 requests per second (!). A number the Cloud was simply not able to cope with.

The number of requests have since been greatly reduced, thanks to @aschempp who released new versions of the Contao Manager. On top of that, I've added caching headers and put the Cloud behind KeyCDN to reduce the number of requests actually reaching the Cloud.

However, there was another issue introduced with autoscaling which caused loads of jobs to be dropped and thus seemed like they were running endlessly. Today, I've disabled autoscaling and fixed the number of workers for Contao to 7 so this issue should be gone for the moment. Due to technical reasons we cannot increase the number to more than 7 for now.

@leofeyer and I are actively working on moving the Composer Resolver Cloud away from Google Cloud and we'll go back to some good old regular, powerful VPS instead of some fancy Kubernetes autoscaling stuff.
This is a bigger task and might well take another month or two to complete, though.

Moreover, I've been actively working on improving the performance of Composer v2 which will hopefully allow us to run even more workers with the same amount of resources or in best case, even forgo the Resolver Cloud completely.

I'll post here once migration is done. Until then, please bear with the lower number of workers.

@Toflar
Copy link
Member Author

Toflar commented Oct 16, 2020

Because a few people were asking (Numbers on the left represent the total number of jobs since we started counting in March 2019):

graph

@Toflar
Copy link
Member Author

Toflar commented Oct 23, 2020

Well, we were pretty quick and worked hard to get this issue resolved asap. Hosting provider migration will take place next Tuesday: https://composer-resolver-cloud.statuspage.io/incidents/pd93jbwqxzdj

Contao will be back at 10 workers ready for your projects.

Next up will be Composer 2.0 preparation work where I expect us being able to double the workers to 20.
Future Composer 2 versions will likely become even more memory efficient and thus allowing us to increase that number even more. I know that because I work on it myself :P

@Toflar
Copy link
Member Author

Toflar commented Oct 27, 2020

Migration's been completed successfully. Goodbye Google and k8s for now 👋

@Toflar Toflar closed this as completed Oct 27, 2020
@aschempp aschempp unpinned this issue Oct 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant