Workers fail with 401

[sigurdm]

Workers sometimes fail with:

Error:
    RequestException(status = 401)
    {"error":{"code":"MissingAuthentication","message":"Authentication is required, please add `authorization` header."},"code":"MissingAuthentication","message":"Authentication is required, please add `authorization` header."}

Unclear what is going on. Are we really missing the header?
Can we handle this more gracefully?