Times out, but travis succeeded

[carols10cents]

On this crates.io PR (and not any other PRs today), the homu checks have timed out multiple times, even though the auto branch has built successfully in travis.

1. I have no idea what's going wrong
2. I don't see anything in the homu commands that will cancel this situation?

[jtgeibel]

I think I'm running into this on rust-lang/crates.io#1956 as well. The tests have passed, and it has not yet timed out.

[carols10cents]

Well now the queue looks like this:

That is, there are approved PRs, but no pending PRs-- bors isn't doing anything :( #1964 did time out again and is at the bottom under failed.

[pietroalbini]

A PR in the Cargo repository (rust-lang/cargo#7547) messed up humu's internal state, triggering an internal assertion. Now that I synchronized that repository everything started flowing again.

Dec 19 09:37:03 localhost homu: Traceback (most recent call last):                                                                             
Dec 19 09:37:03 localhost homu:   File "/usr/local/bin/bottle.py", line 862, in _handle                                                                   
Dec 19 09:37:03 localhost homu:     return route.call(**args)                                                                                  
Dec 19 09:37:03 localhost homu:   File "/usr/local/bin/bottle.py", line 1742, in wrapper                                                                  
Dec 19 09:37:03 localhost homu:     rv = callback(*a, **ka)                                                                                    
Dec 19 09:37:03 localhost homu:   File "/homu/homu/server.py", line 576, in github                                                                        
Dec 19 09:37:03 localhost homu:     g.queue_handler()                                                                                          
Dec 19 09:37:03 localhost homu:   File "/homu/homu/main.py", line 1795, in queue_handler                                                                  
Dec 19 09:37:03 localhost homu:     return process_queue(states, repos, repo_cfgs, logger, buildbot_slots, db, git_cfg)  # noqa                
Dec 19 09:37:03 localhost homu:   File "/homu/homu/main.py", line 1416, in process_queue                                                                  
Dec 19 09:37:03 localhost homu:     logger, db, git_cfg):                                                                                      
Dec 19 09:37:03 localhost homu:   File "/homu/homu/main.py", line 1395, in start_build_or_rebuild                                                         
Dec 19 09:37:03 localhost homu:     return start_build(state, repo_cfgs, *args)                                                                
Dec 19 09:37:03 localhost homu:   File "/homu/homu/main.py", line 1206, in start_build                                                                    
Dec 19 09:37:03 localhost homu:     assert state.head_sha == pr.head.sha                                                                       
Dec 19 09:37:03 localhost homu: AssertionError                                                                                                            

I don't feel like closing the issue because the problem might happen again, but I don't have the time to track it down and fix it.

[carols10cents]

Thank you Pietro!!

[smarnach]

I think we saw two separate issues:

• Homu stopped triggering builds for many PRs, so after "bors r+" nothing really happened. This affected all PRs for a while, and is now fixed again.

• For some PRs, Homu detects timeouts while the build actually succeeded long before:

  • Bump babel-plugin-transform-object-rest-spread from 6.23.0 to 6.26.0 crates.io#2010 (comment)
  • [Security] Bump lodash.template from 4.4.0 to 4.5.0 crates.io#1964 (comment)

  This problem only affects a few PRs, and it still persists, so I believe it is unrelated.

[smarnach]

The problem with timeouts seems to be a bug in Travis – see the discussion on rust-lang/crates.io#1964. I will contact Travis support.

[smarnach]

I haven't heard back from Travis support so far. I will update this ticket as soon as I get a reply.

[smarnach]

@carols10cents figured out how this happened (well done!)

Travis support replied with an answer to the effect of "well, this is indeed puzzling" a few days ago. I will update them and tell them to fix that bug. :)

This issue can be closed now (but I don't have permisison to close it myself).

[smarnach]

Maybe there is indeed a bug in Homu after all.

If the commit message of the commit Homu is testing contains one of the commands to skip the CI build, Homu will always time out trying to test the commit, even after the Travis bug that results in the previous commit being built is fixed. So it would be reasonable to strip these commands from the merge commit message on the auto branch.

(Relevant code that would need to be changed)