Is that possible to run job which takes more than 6 hours on self-hosted runner?

[victorx-deckard]

In my CI/CD setup, one step is to retrain a deep learning model. We setup our own GPU-box as a self-hosted runner to run this step. However, the training takes about one day, which is much longer than the current 6-hours cap GitHub Actions allowed. Is there any way to workaround this limitation? 

[Yanjingzhu]

Yes, you can use self-hosted runner to run more than 6 hours job.

• Self-hosted runners give you the opportunity to persist whatever you like for your jobs and not be subject to the six-hour time-out in hosted virtual environments.

Please refer to this official blog:

https://github.blog/2019-11-05-self-hosted-runners-for-github-actions-is-now-in-beta/#advantages-of-self-hosted-runners

[victorx-deckard]

This is interesting. My test shows the jos is killed after 6 hours even running on self-hosted runner. 

Here is a snapshot of the summary of the run. As it shows, the job was killed after 6 hours.

Screenshot from 2019-12-06 11-03-48.png1475×361 32.6 KB

Just in case one might suspect the job had not been run on self-hosted runner. Here is the screenshot of the shell where I run the self-hosted runner agent. As can be seen from the timestamps, the job (the 3rd and 4th ones) were killed after 6 hours.

Screenshot from 2019-12-06 11-05-01.png987×330 62.7 KB

[Yanjingzhu]

The maximum number of minutes to let a job run before GitHub automatically cancels it. In default is 360 minutes. 

Please try to specify a larger minutes to timeout-minutes under your job. 

https://help.github.com/en/actions/automating-your-workflow-with-github-actions/workflow-syntax-for-github-actions#jobsjob_idtimeout-minutes

[Jianlin-lv]

I was very confused. It didn’t work for me.  

Even “Both jobs. <job_id> .steps.timeout-minutes” and “jobs. <job_id> .timeout-minutes” be set to 7200,  The jobs was still canceled by GitHub automatically.

My YAML file:

https://github.com/Jianlin-lv/buildx-demo/blob/master/.github/workflows/build_container.yml

https://github.com/Jianlin-lv/buildx-demo/actions/runs/105486327

[DavidGOrtega]

Solution is false.
Jobs can not run longer than six hours by any meaning

[ashipanw]

+1 . Can some one please update if we can run any jobs more than 6hrs ?

[ashipanw]

Just to let all the folks know, Yes its possible to run job/steps for more than 6 hrs , provided you are using self hosted runners .for ex: Use job-timeout: 1000 parameter to allow jobs/steps to run for 1000 mins in your workflow.

Cheers !!

[acevos01]

Can someone help with this? This solution doesn’t work. I’m still getting a 6h timeout even after putting timeout-minutes in the step and job level with a self hosted runner. As aside note I’m using reusable workflows (reusable workflows can’t have timeout-minutes property)

[ptc-aramachandra]

Do we have any updates on the solution. Self hosted runners even with timeout parameter being more than 6 hours doesnt work. It still cancels the job after 6 hours. Any solutions here ?

[nullsurface]

I'm running into the same issue with a self-hosted runner.

[tt-rkim]

Still running into this. Any movement on allowing longer than 360 min jobs on self hosted runners? We would like to run stress tests on custom hardware, which shouldn't affect extra runner infrastructure GitHub makes for Actions.

[XargonWan]

I can report that still the limit is in place and I'm unable to do a longer build beyond 6h.

[XargonWan]

I can report that still the limit is in place and I'm unable to do a longer build beyond 6h.

[JohnDalyIonQ]

Thanks for saying something about this, it's an extremely frustrating problem to hit, especially if you have to wait 6 hours to see if you fixed it!

[tt-rkim]

We were actually able to get longer than 6h builds on self-hosted runners recently with something like the following:

jobs:
  stress-build-and-unit-tests:
    timeout-minutes: 1440
    runs-on: "self-hosted"
    steps:
      - ...

The key seems to be set to it at the job level, not step level

[chintanp]

@tt-rkim what is the maximum timeout value have you been able to test this for? One of the limits is 35 days, I wonder if this is able to get to that or cross that limit.

[tt-rkim]

We have been able to get up to 25 hours, unfortunately no more than 35 days.

I am unsure how to get around the total workflow time... unfortunately my suspicion is because the workflow is a GitHub-managed construct in their architecture or whatever, this doesn't sound like something you could get around. Or at least, I have no ideas currently on how to .