Make upload more robust by ignoring spurious errors while polling the scan status.

[vilagithub]

… scan status.

While the snap is scanned on the store, the status is polled until completion.

If an error occurs while acquiring the status, ignore the errors.

Those errors are rare and the next retry generally succeeds so ignoring those errors is harmless.

If they persist until the last retry, something else may be going on and this will still be reported.

LP: #1572963

[snappy-m-o]

Can one of the admins verify this patch?

[snappy-m-o]

Can one of the admins verify this patch?

[sergiusens]

What was HTTPError, will we loop forever if the resource does not exist?

[vilagithub]

What was HTTPError

I don't remember encountering it in this particular case but the code involved can trigger it. I can remove it if you prefer.

will we loop forever if the resource does not exist?

No. The caller retries a fixed number of times and will report the last error.

[elopio]

According to [1], we should return an exception instead of None. That makes sense because the name of the exception will make it's meaning clearer than None.

[1] https://www.goodreads.com/book/show/23020812-effective-python

[vilagithub]

we should return an exception instead of None.

No, that's the bug. The caller (intermediate, hidden, out of our reach) can't catch it nor act properly. Returning None tells the caller: nah, not completed yet.

[elopio]

I see, @retry doesn't let you handle the exception. The comments make this clear, so I'm ok with this.

[elopio]

I think I would name these helper funcions with a leading underscore, to make it clear that they are called only from higher level functions.

[vilagithub]

There is a refactor in progress in another branch, I'd rather address that there ?

[elopio]

works for me.

[elopio]

this looks good to me. It's good that you moved the func out of common, because it was called only once.
I would prefer not to return None to signal the problem.

[elopio]

retest this please

[vilagithub]

From http://162.213.35.179:8080/job/github-snapcraft-autopkgtest-cloud/597/console
adt-run [16:32:03]: version 3.20.3
...
adt-run [17:02:03]: ERROR: testbed failure: cannot send to testbed: ['BrokenPipeError: [Errno 32] Broken pipe\n']

So 30 minutes hanging on the instance creation. Re-test ?
3.20.4 is out for xenial but I don't think that will fix that issue.

From http://162.213.35.179:8080/job/github-snapcraft-examples-tests-cloud/697/console

2016/04/27 16:31:55 *** Creating a Snappy m1.medium instance for release rolling, channel edge and image type custom ***

goroutine 1 [running]:
log.Panicf(0x5b1d20, 0x36, 0xc820039e58, 0x3, 0x3)
/usr/lib/go/src/log/log.go:327 +0xd8
main.main()
/build/snappy-tests-job-LsZciy/snappy-tests-job-1.0.0ubuntu1/obj-x86_64-linux-gnu/src/launchpad.net/snappy-tests-job/cmd/snappy-cloud-client/main.go:68 +0xb93

[sergiusens]

retest this

[elopio]

OK to test

[kyrofa]

Looks good to me 👍