Simple task to finish inactive builds #3312

humitos · 2017-11-23T22:46:05Z

This is an initial idea to solve these kind of issues and avoid user confusions:

Fixes Documentation stuck in 'cloning' state #2795
Fixes Several recent consecutive builds are stuck in "Building" state #3308
Fixes 3 week old Build Stuck Cloning #3126
Fixes Cleanup misreported failed builds #3230
Related Allow user to kill a build #2471

I found that we are not running celery beat for periodic tasks so we need to define how this will be executed. Maybe it could be done as a django command and ran it by cron (depending how is the current setup at RTD servers)
We should consider the custom time limit for some project, since if it's more than 45 minutes this task will mark them as finished
Define how we want to show this custom error

Here is an example on how this will look like

agjohnson · 2017-11-27T17:40:39Z

Ah yes, this could be updated in our docs probably.

We run celery beat just as we run celery:

celery -A readthedocs worker -B

So this would be another command to run for local development. You want to run on a separate worker, as it should only have 1 concurrent process. That is, this will result in multiple scheduled tasks, as there is 4 concurrency:

celery -A readthedocs worker -Q default,celery,web,builder -B

I'm fine not supporting this when running celery as ALWAYS_EAGER -- maybe throwing a warning through a Django system check?

Custom time limit should be 150-200% of the project build time. The project build time is the container_time_limit or the default docker time limit. The celery task time limit is 120% of the project build time, I believe. So 150-200% should extend past the timeout on celery and docker.

Also, thinking more on this, it might be helpful to include more information in the error message. Adding something similar to our general error reporting would help -- "If you continue to encounter this error, file a support request with and reference this build id (#)"

agjohnson

I think this is in the right direction. I added some feedback on the questions you've had so far, so we'll probably need another review when those changes are added.

agjohnson · 2017-11-27T17:45:23Z

readthedocs/projects/tasks.py

+    for build in Build.objects.filter(query):
+        build.success = False
+        build.state = BUILD_STATE_FINISHED
+        build.error = 'This build was terminated due to inactivity.'


On my PR, I'm accumulating these error messages in one spot, which might make sense. I'm storing these error messages all in a central projects.exceptions file. This could be implemented as an exception, but I'm not sure if the exception will be raised otherwise -- we could combine the message for this and the timeout kill though.

agjohnson · 2017-11-27T17:45:55Z

readthedocs/projects/tasks.py

+    query = (~Q(state=BUILD_STATE_FINISHED) &
+             Q(date__lte=datetime.datetime.now() - datetime.timedelta(minutes=45)))
+    # TODO: consider ``poject.container_time_limit`` since it could be bigger than 45 minutes
+    for build in Build.objects.filter(query):


This could be build.update() instead.

If we are going to use update we can't put the build id # in the message.

So, for now, I'm adding this message:

error=('This build was terminated due to inactivity. If you continue ' 'to encounter this error, file a support request.')

Also, I think I will limit the query to the first 25 or 50 elements because otherwise the first time this task will be ran it will take tons of existing builds from all the times...

Besides, this is not valid...

AssertionError: Cannot update a query once a slice has been taken.

So, maybe go back to my approach is better.

safwanrahman · 2017-11-27T17:51:33Z

I think we should use celery beat for this. otherwise this task does not have any benifit

agjohnson · 2017-11-28T00:21:52Z

Celery beat is going to be used. The task above is will be a scheduled task

humitos · 2017-12-05T21:38:35Z

I just update this PR with:

do not use update since I want an slice of the filter and also we want to the build.pk inside of the message
1:30hs as a grace time to the build to finish (I think it's enough) --I tried to make use of the project__container_time_limit but it was complicated and I wasn't able to end up with the correct query
add a log.info with the information about how many builds were marked as "Terminated..."

Let me know if this is OK for you and also how to configure this as an scheduled task since I don't find the CELERY BEAT setting.

humitos · 2017-12-05T21:39:26Z

We can always improve this task but it should be enough (for some definition) at this moment. It could be a good starting point.

agjohnson

So this method is close. Perhaps my suggestion is the best compromise between complexity and the most correct UX.

agjohnson · 2017-12-07T01:57:25Z

readthedocs/projects/tasks.py

+
+@app.task()
+def finish_inactive_builds():
+    delta = datetime.timedelta(hours=1, minutes=30)


Lets use a timedelta of our base task timeout. For an example of how we add overhead to this, see:
https://github.com/rtfd/readthedocs.org/blob/master/readthedocs/core/utils/__init__.py#L118-L128

There's a second part to this though, see below.

agjohnson · 2017-12-07T02:00:24Z

readthedocs/projects/tasks.py

+
+    builds = Build.objects.filter(query)[:50]
+    for build in builds:
+        build.success = False


Let's check for the timeout date here, ie - if build.date + build.project.container_time_limit > now: continue.

So the query will surface extra builds, but the custom timeout builds will be an exception to this case. They keep running through this loop until the condition above is true.

Does this seem reasonable?

Yes. I think this is the easiest and more reasonable thing to do.

"I was overengeering it" as Eric said :)

agjohnson · 2017-12-07T02:01:04Z

readthedocs/projects/tasks.py

+            'This build was terminated due to inactivity. If you '
+            'continue to encounter this error, file a support '
+            'request with and reference this build id ({0}).'.format(build.pk)
+        )


This should be translated, though I'm not sure if we're equipped to actually translate this yet.

Equipped? In what sense?

I don't understand how/where the translation are working yet.

humitos · 2017-12-07T16:29:48Z

This PR should be ready.

Do I need to do something in particular to enable this task on celery beat?

Fixes #2795 Fixes #3308

agjohnson

Looks good!

I think we'll want at least 3 test cases out of this:

Legitimate build isn't marked finished early
Build is custom time limit isn't marked finished early
Build without custom time limit is marked finished

agjohnson · 2017-12-11T07:50:12Z

readthedocs/projects/tasks.py

+
+        if build.project.container_time_limit:
+            custom_delta = datetime.timedelta(
+                minutes=int(build.project.container_time_limit))


container_time_limit is in seconds, so delta will need to be changed here

humitos · 2017-12-12T16:26:38Z

I added a test case for this escenario. I thought it was better to have 3 builds at the same time instead of one build per test. Let me know if you agree with me or want me to split them.

I think it's easier to understand and also it's more close to the reality.

agjohnson · 2017-12-14T20:58:25Z

Giddyup. The tests look great, I don't think it's important to split them up at all

humitos · 2017-12-14T21:05:01Z

The only thing missing is setup the celery beat to effectively run this task El 14 dic. 2017 3:59 p. m., "Anthony" <notifications@github.com> escribió:

…

Merged #3312 <#3312>. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#3312 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAO7sF-eg9wRiH5T-5CALAU1bL2qOBC_ks5tAYwkgaJpZM4QpOay> .

humitos · 2017-12-23T04:52:09Z

@agjohnson @ericholscher what do we need to do to enable this task to be ran by celery beat?

ericholscher · 2017-12-24T15:14:53Z

@humitos https://github.com/readthedocs/readthedocs-ops/blob/master/salt/base/readthedocs/site/settings.py#L132

humitos added the PR: work in progress Pull request is not ready for full review label Nov 23, 2017

humitos requested a review from agjohnson November 23, 2017 22:46

humitos added the Needed: design decision A core team decision is required label Nov 23, 2017

agjohnson reviewed Nov 27, 2017

View reviewed changes

agjohnson removed the Needed: design decision A core team decision is required label Nov 27, 2017

agjohnson assigned humitos Dec 1, 2017

agjohnson reviewed Dec 7, 2017

View reviewed changes

humitos added PR: ready for review and removed PR: work in progress Pull request is not ready for full review labels Dec 7, 2017

Simple task to finish inactive builds

a014493

Fixes #2795 Fixes #3308

humitos force-pushed the humitos/inactive-build-cleanup branch from c15d774 to a014493 Compare December 7, 2017 17:36

agjohnson reviewed Dec 11, 2017

View reviewed changes

humitos added 2 commits December 12, 2017 10:34

Fix time unit

adc6634

Test for finish_inactive_builds task

6532ee7

Fix lint

859c305

agjohnson merged commit a700bbd into master Dec 14, 2017

agjohnson deleted the humitos/inactive-build-cleanup branch December 14, 2017 20:59

humitos mentioned this pull request Jan 22, 2018

Builds get stuck in "Cloning" state #2047

Closed

humitos mentioned this pull request Mar 21, 2018

Allow user to kill a build #2471

Closed

humitos mentioned this pull request Aug 15, 2018

Protect against web/api 5xx responses #2258

Closed

humitos mentioned this pull request Oct 20, 2020

Improve Handling on multiple commits #7576

Closed

humitos mentioned this pull request Nov 12, 2020

Feature Request: Allow maintainers to cancel PR builds #7660

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Simple task to finish inactive builds #3312

Simple task to finish inactive builds #3312

humitos commented Nov 23, 2017 •

edited by agjohnson

agjohnson commented Nov 27, 2017

agjohnson left a comment

agjohnson Nov 27, 2017

agjohnson Nov 27, 2017

humitos Dec 5, 2017

humitos Dec 5, 2017

humitos Dec 5, 2017

safwanrahman commented Nov 27, 2017

agjohnson commented Nov 28, 2017

humitos commented Dec 5, 2017

humitos commented Dec 5, 2017

agjohnson left a comment

agjohnson Dec 7, 2017

agjohnson Dec 7, 2017

humitos Dec 7, 2017

agjohnson Dec 7, 2017

humitos Dec 7, 2017

humitos commented Dec 7, 2017

agjohnson left a comment

agjohnson Dec 11, 2017

humitos commented Dec 12, 2017

agjohnson commented Dec 14, 2017

humitos commented Dec 14, 2017 via email

humitos commented Dec 23, 2017

ericholscher commented Dec 24, 2017

Simple task to finish inactive builds #3312

Simple task to finish inactive builds #3312

Conversation

humitos commented Nov 23, 2017 • edited by agjohnson

agjohnson commented Nov 27, 2017

agjohnson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

safwanrahman commented Nov 27, 2017

agjohnson commented Nov 28, 2017

humitos commented Dec 5, 2017

humitos commented Dec 5, 2017

agjohnson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos commented Dec 7, 2017

agjohnson left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

humitos commented Dec 12, 2017

agjohnson commented Dec 14, 2017

humitos commented Dec 14, 2017 via email

humitos commented Dec 23, 2017

ericholscher commented Dec 24, 2017

humitos commented Nov 23, 2017 •

edited by agjohnson