Skip to content

Improve tool output upon remote server errors #2

@athos-ribeiro

Description

@athos-ribeiro

When services return errors, those errors should not be dumped in the tool output (at least not to stdout). Instead, we can have a short message output in the section for the service that failed carrying the status code and a short version of the error message. Optionally, the full error message can be dumped to STDERR (maybe through a verbose or debug flag).

Here is the output I got during an outage + hitting GH rate limits:

Triage 2026-05-11 for team 'ubuntu-server'
Items updated on 2026-05-11 (Monday)...

The authorization page:
 (https://launchpad.net/+authorize-token?oauth_token=REDACTED_TOKEN&allow_permission=DESKTOP_INTEGRATION)
should be opening in your browser. Use your browser to authorize
this program to access Launchpad on your behalf.
Waiting to hear from Launchpad about your decision...
## Proposed Migration (30 packages)  [generated 2026-05-12 06:15 UTC]
filter: teams=ubuntu-server
  Package              | Old → New                                         | Since      | Notes
�[0;31m■�[0m clamav | 1.4.3+dfsg-2ubuntu3 → 1.4.4+dfsg-0ubuntu0.26.04.1 | 2026-04-30 | [autopkgtest]
�[0;31m■�[0m gdk-pixbuf | 2.44.5+dfsg-4ubuntu1 → 2.44.6+dfsg-2              | 2026-04-30 | [autopkgtest]
�[0;31m■�[0m haproxy | 3.2.9-1ubuntu2 → 3.2.9-1ubuntu2.1                 | 2026-04-30 | [autopkgtest]
�[0;31m■�[0m openssh | 1:10.2p1-2ubuntu3 → 1:10.2p1-2ubuntu3.2           | 2026-04-30 | [autopkgtest]
�[0;31m■�[0m strongswan | 6.0.4-1ubuntu2 → 6.0.4-1ubuntu3                   | 2026-04-30 | [autopkgtest]
�[0;31m■�[0m libarchive | 3.8.5-1ubuntu2 → 3.8.7-1                          | 2026-04-30 | [autopkgtest]
�[0;31m■�[0m libpng1.6 | 1.6.57-1 → 1.6.58-1                               | 2026-05-03 | [autopkgtest]
�[0;31m■�[0m policykit-1 | 127-2ubuntu1 → 127-3                              | 2026-05-03 | [autopkgtest]
�[0;31m■�[0m python-urllib3 | 2.6.3-1ubuntu1 → 2.6.3-2                          | 2026-05-03 | [autopkgtest]
�[0;31m■�[0m xdp-tools | 1.6.2-1ubuntu1 → 1.6.3-1                          | 2026-05-03 | [autopkgtest]
�[0;31m■�[0m apt | 3.2.0 → 3.3.0                                     | 2026-05-04 | [autopkgtest, depends]
�[0;31m■�[0m exim4 | 4.99.1-1ubuntu1 → 4.99.1-1ubuntu1.1               | 2026-05-04 | [autopkgtest]
�[0;31m■�[0m openvpn | 2.7.0-1ubuntu1 → 2.7.3-1ubuntu1                   | 2026-05-04 | [autopkgtest]
�[0;31m■�[0m qemu | 1:10.2.1+ds-1ubuntu3 → 1:10.2.1+ds-1ubuntu4       | 2026-05-05 | [autopkgtest]
�[0;31m■�[0m glib2.0 | 2.88.0-1 → 2.88.1-2                               | 2026-05-05 | [autopkgtest]
�[0;31m■�[0m python-django | 3:5.2.9-0ubuntu4 → 3:5.2.9-0ubuntu4.1             | 2026-05-05 | [autopkgtest]
�[0;31m■�[0m autoconf | 2.72-3.1ubuntu2 → 2.72-6ubuntu1                   | 2026-05-05 | [autopkgtest]
�[0;33m■�[0m corosync | 3.1.9-2ubuntu3 → 3.1.10-2ubuntu1                  | 2026-05-05 | LP#1921137  [autopkgtest]
�[0;31m■�[0m curl | 8.18.0-1ubuntu2 → 8.20.0-1ubuntu1                 | 2026-05-05 | [autopkgtest]
�[0;31m■�[0m inetutils | 2:2.7-2ubuntu1 → 2:2.8-1ubuntu1                   | 2026-05-05 | [autopkgtest]
�[0;31m■�[0m rabbitmq-server | 4.0.5-10ubuntu5 → 4.0.5-14ubuntu1                 | 2026-05-05 | [autopkgtest]
�[0;31m■�[0m ruby-ethon | 0.16.0-3ubuntu2 → 0.18.0-2                        | 2026-05-05 | [autopkgtest]
�[0;31m■�[0m ruby-rack | 3.2.4-1ubuntu1 → 3.2.6-2                          | 2026-05-05 | [autopkgtest]
�[0;31m■�[0m vim | 2:9.1.2141-1ubuntu4 → 2:9.2.0428-1ubuntu1         | 2026-05-05 | [autopkgtest, missingbuild]
�[0;31m■�[0m dbus | 1.16.2-2ubuntu4 → 1.16.2-4ubuntu1                 | 2026-05-06 | [autopkgtest]
�[0;31m■�[0m pipewire | 1.6.2-1ubuntu1 → 1.6.4-1ubuntu1                   | 2026-05-06 | [autopkgtest]
�[0;31m■�[0m screen | 4.9.1-3ubuntu2 → 5.0.1-1.1                        | 2026-05-06 | [autopkgtest]
�[0;31m■�[0m ethtool | 1:6.19-1 → 1:7.0-1                                | 2026-05-06 | [autopkgtest]
�[0;31m■�[0m apache2 | 2.4.66-2ubuntu2 → 2.4.66-2ubuntu2.1               | 2026-05-06 | [autopkgtest]
�[0;31m■�[0m postfix | 3.10.6-4ubuntu2 → 3.10.6-4ubuntu2.1               | 2026-05-07 | [autopkgtest]

## Discourse (0 topics)


Error fetching 'launchpad':
Traceback (most recent call last):
  File "PROJECT_ROOT/startriage/triage.py", line 176, in _await_and_print
    result: TriageResult = await task
                           ^^^^^^^^^^
  File "PROJECT_ROOT/startriage/sources/launchpad/triage.py", line 288, in find
    lp_tasks = await asyncio.to_thread(
               ^^^^^^^^^^^^^^^^^^^^^^^^
    ...<8 lines>...
    )
    ^
  File "/usr/lib/python3.13/asyncio/threads.py", line 25, in to_thread
    return await loop.run_in_executor(None, func_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
  File "PROJECT_ROOT/startriage/sources/launchpad/finder.py", line 337, in fetch_bugs
    expiring_tagged = _expiring_window(expire_level1_days)
  File "PROJECT_ROOT/startriage/sources/launchpad/finder.py", line 307, in _expiring_window
    for t in _search_tasks_all_series(
             ~~~~~~~~~~~~~~~~~~~~~~~~^
        ubuntu,
        ^^^^^^^
    ...<2 lines>...
        status=OPEN_BUG_STATUSES,
        ^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "PROJECT_ROOT/startriage/sources/launchpad/finder.py", line 71, in _search_tasks_all_series
    result = {(task.bug_link, _fast_target_name(task)): task for task in distro.searchTasks(*args, **kwargs)}
                                                                         ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/athos/.virtualenvs/startriage/lib/python3.13/site-packages/lazr/restfulclient/resource.py", line 642, in __call__
    response, content = self.root._browser._request(
                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        url,
        ^^^^
    ...<2 lines>...
        extra_headers=extra_headers,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "$HOME/.virtualenvs/startriage/lib/python3.13/site-packages/lazr/restfulclient/_browser.py", line 484, in _request
    raise error
lazr.restfulclient.errors.ServerError: HTTP Error 503: Service Unavailable
Response headers:
---
-content-encoding: gzip
connection: close
content-length: 10380
content-type: text/html;charset=utf-8
date: Tue, 12 May 2026 07:57:34 GMT
retry-after: 900
server: gunicorn
status: 503
transfer-encoding: chunked
vary: Accept-Encoding
x-lazr-oopsid: OOPS-c410218fca12b647b51d880c16fd0f89
x-powered-by: Zope (www.zope.org), Python (www.python.org)
x-request-id: ID
x-vcs-revision: 91d5f4cbcbec62cc98ca9991c8ad99af4e21f85a
---
Response body:
---
b'<!DOCTYPE html>\n<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en" dir="ltr">\n  <head>\n    <meta charset="UTF-8" />\n    <title>Error: Timeout</title>\n    <link rel="apple-touch-icon" sizes="180x180" href="/@@/apple-touch-icon.png?v=2022" />\n    <link rel="icon" type="image/png" sizes="32x32" href="/@@/favicon-32x32.png?v=2022" />\n    <link rel="icon" type="image/png" sizes="16x16" href="/@@/favicon-16x16.png?v=2022" />\n    <link rel="manifest" href="/@@/site.webmanifest?v=2022" />\n    <link rel="mask-icon" href="/@@/safari-pinned-tab.svg?v=2022" color="#e9531f" />\n    <link rel="shortcut icon" href="/@@/favicon.ico?v=2022" />\n    <meta name="msapplication-TileColor" content="#da532c" />\n    <meta name="msapplication-config" content="/@@/browserconfig.xml?v=2022" />\n    <meta name="theme-color" content="#ffffff" />\n    \n    \n    \n\n    \n  \n  <link type="text/css" rel="stylesheet" media="screen, print" href="/+icing/rev91d5f4cbcbec62cc98ca9991c8ad99af4e21f85a/combo.css" />\n\n\n    \n\n    \n      \n      \n    \n\n    \n    \n\n    \n\n    \n  \n\n  \n  \n  <script type="text/javascript">\n    var LP = {\n        cache: {},\n        links: {}\n    };\n  </script>\n\n  \n\n  <script type="text/javascript">var cookie_scope = \'; Path=/; Secure; Domain=.launchpad.net\';</script>\n\n   <script type="text/javascript" src="/+combo/rev91d5f4cbcbec62cc98ca9991c8ad99af4e21f85a/?yui/yui/yui-min.js&amp;lp/meta.js&amp;yui/loader/loader-min.js"></script>\n   <script type="text/javascript">\n        var raw = null;\n        if (LP.devmode) {\n           raw = \'raw\';\n        }\n        YUI.GlobalConfig = {\n            combine: true,\n            comboBase: \'/+combo/rev91d5f4cbcbec62cc98ca9991c8ad99af4e21f85a/?\',\n            root: \'yui/\',\n            filter: raw,\n            debug: false,\n            fetchCSS: false,\n            maxURLLength: 2000,\n            groups: {\n                lp: {\n                    combine: true,\n                    base: \'/+combo/rev91d5f4cbcbec62cc98ca9991c8ad99af4e21f85a/?lp/\',\n                    comboBase: \'/+combo/rev91d5f4cbcbec62cc98ca9991c8ad99af4e21f85a/?\',\n                    root: \'lp/\',\n                    // comes from including lp/meta.js\n                    modules: LP_MODULES,\n                    fetchCSS: false\n                }\n            }\n        }</script>\n\n  <script type="text/javascript">\n      // we need this to create a single YUI instance all events and code\n      // talks across. All instances of YUI().use should be based off of\n      // LPJS instead.\n      var LPJS = new YUI();\n  </script>\n\n\n\n    <script id="base-layout-load-scripts" type="text/javascript">\n        //<![CDATA[\n        LPJS.use(\'base\', \'node\', \'console\', \'event\',\n            \'oop\', \'lp\', \'lp.app.foldables\',\'lp.app.sorttable\',\n            \'lp.app.inlinehelp\', \'lp.app.links\',\n            \'lp.bugs.bugtask_index\', \'lp.bugs.subscribers\',\n            \'lp.app.ellipsis\', \'lp.code.branchmergeproposal.diff\',\n            \'lp.views.global\',\n             function(Y) {\n\n            Y.on("domready", function () {\n                var global_view = new Y.lp.views.Global();\n                global_view.render();\n\n                Y.lp.app.sorttable.SortTable.init();\n                Y.lp.app.inlinehelp.init_help();\n                Y.lp.activate_collapsibles();\n                Y.lp.app.foldables.activate();\n                Y.lp.app.links.check_valid_lp_links();\n            });\n\n            Y.on(\'lp:context:web_link:changed\', function(e) {\n                  window.location = e.new_value;\n            });\n        });\n        //]]>\n    </script>\n    <script id="base-helper-functions" type="text/javascript">\n         //<![CDATA[\n        // This code is pulled from lp.js that needs to be available on every\n        // request. Pulling here to get it outside the scope of the YUI block.\n        function setFocusByName(name) {\n            // Focus the first element matching the given name which can be focused.\n            var nodes = document.getElementsByName(name);\n            var i, node;\n            for (i = 0; i < nodes.length; i++) {\n                node = nodes[i];\n                if (node.focus) {\n                    try {\n                        // Trying to focus a hidden element throws an error in IE8.\n                        if (node.offsetHeight !== 0) {\n                            node.focus();\n                        }\n                    } catch (e) {\n                        LPJS.use(\'console\', function(Y) {\n                            Y.log(\'In setFocusByName(<\' +\n                                node.tagName + \' type=\' + node.type + \'>): \' + e);\n                        });\n                    }\n                    break;\n                }\n            }\n        }\n\n        function selectWidget(widget_name, event) {\n          if (event && (event.keyCode === 9 || event.keyCode === 13)) {\n              // Avoid firing if user is tabbing through or simply pressing\n              // enter to submit the form.\n              return;\n          }\n          document.getElementById(widget_name).checked = true;\n        }\n        //]]>\n    </script>\n\n    \n      \n    \n  </head>\n\n  <body id="document" itemscope="" itemtype="http://schema.org/WebPage" class="tab-unknown\n      main_only\n      public\n      yui3-skin-sam">\n          \n          \n          \n    <div class="yui-d0">\n      <div id="locationbar" class="login-logout">\n        \n\n\n<div id="logincontrol">\n  <form action="/+logout" method="post">\n    <input type="hidden" name="loggingout" value="1" />\n    \n    \n    <a href="/~athos" class="sprite person">Athos Ribeiro (athos)</a> &bull;\n    <input type="submit" name="logout" value="Log Out" />\n  </form>\n</div>\n\n\n      </div><!--id="locationbar"-->\n\n      <div id="watermark" class="watermark-apps-portlet">\n        <div>\n          <img alt="" width="64" height="64" src="/@@/launchpad-logo" />\n        </div>\n        <div class="wide">\n          <h2 id="watermark-heading"><span>Launchpad.net</span></h2>\n        </div>\n        \n  <!-- Application Menu -->\n  <ul class="facetmenu">\n    \n      <li class="overview active"><a href="https://launchpad.net/ubuntu">Overview</a></li>\n      \n      \n    \n    \n      \n      <li class="branches"><a href="https://code.launchpad.net/ubuntu">Code</a></li>\n      \n    \n    \n      \n      <li class="bugs"><a href="https://bugs.launchpad.net/ubuntu">Bugs</a></li>\n      \n    \n    \n      \n      <li class="specifications"><a href="https://blueprints.launchpad.net/ubuntu">Blueprints</a></li>\n      \n    \n    \n      \n      <li class="translations"><a href="https://translations.launchpad.net/ubuntu">Translations</a></li>\n      \n    \n    \n      \n      <li class="answers"><a href="https://answers.launchpad.net/ubuntu">Answers</a></li>\n      \n    \n  </ul>\n\n      </div>\n\n      \n        <div id="maincontent" class="yui-main">\n          <div class="yui-b" dir="ltr">\n            <div class="context-publication">\n              \n              \n\n              <div id="registration" class="registering">\n                \n              </div>\n            </div>\n\n            \n            <div id="request-notifications">\n              \n            </div>\n\n            \n              <div class="top-portlet">\n      <h1 class="exception">Timeout error</h1>\n      <p>\n        Sorry, something just went wrong in Launchpad.\n      </p>\n      <p>\n        We&#8217;ve recorded what happened,\n        and we&#8217;ll fix it as soon as possible.\n        Apologies for the inconvenience.\n      </p>\n      <p>\n        Trying again in a couple of minutes might work.\n      </p>\n      <p>\n        If you report this as a bug, please include the error ID below,\n        preferably by copying and pasting it rather than by taking a\n        screenshot.\n      </p>\n      <p>\n        (Error <abbr>ID</abbr>:\n        <code class="oopsid">OOPS-c410218fca12b647b51d880c16fd0f89</code>)\n      </p>\n      \n    </div>\n            \n            \n          </div><!-- yui-b -->\n        </div><!-- yui-main -->\n\n        \n          <!-- yui-b side -->\n        \n      <!-- yui-t4 -->\n\n      \n  <div id="footer" class="footer">\n    <div class="lp-arcana">\n        <div class="lp-branding">\n          <a href="https://launchpad.net/"><img src="/@@/launchpad-footer-logo.svg" alt="Launchpad" width="65" height="18" /></a>\n          &nbsp;&bull;&nbsp;\n          <a href="https://launchpad.net/+tour">Take the tour</a>\n          &nbsp;&bull;&nbsp;\n          <a href="https://documentation.ubuntu.com/launchpad/">Read the guide</a>\n          &nbsp;\n          <form id="globalsearch" method="get" accept-charset="UTF-8" action="https://launchpad.net/+search">\n            <input type="search" id="search-text" name="field.text" />\n            <input type="image" src="/@@/search" style="vertical-align:5%" alt="Search Launchpad" />\n          </form>\n        </div>\n        \n  \n\n    </div>\n\n    <div class="colophon">\n      &copy; 2004\n      <a href="http://canonical.com/">Canonical&nbsp;Ltd.</a>\n      &nbsp;&bull;&nbsp;\n      <a href="https://ubuntu.com/legal/launchpad-terms-of-service">Terms of use</a>\n      &nbsp;&bull;&nbsp;\n      <a href="https://www.ubuntu.com/legal/dataprivacy">Data privacy</a>\n      &nbsp;&bull;&nbsp;\n      \n      <a href="/support">Contact Launchpad Support</a>\n      &nbsp;&bull;&nbsp;\n      <a href="http://blog.launchpad.net/">Blog</a>\n      \n\t&nbsp;&bull;&nbsp;\n\t<a href="https://canonical.com/careers">Careers</a>\n      \n      &nbsp;&bull;&nbsp;\n      <a href="https://ubuntu.social/@launchpadstatus">System status</a>\n      <span id="lp-version">\n      &nbsp;&bull;&nbsp;\n        91d5f4c\n        \n        \n        (<a href="https://documentation.ubuntu.com/launchpad/developer/how-to/get-source-code/">Get the code!</a>)\n      </span>\n    </div>\n  </div>\n\n    </div><!-- yui-d0-->\n\n    \n  \n  \n  \n  <script id="json-cache-script">LP.cache = {"related_features": {}};</script>\n\n    \n  \n\n    \n  </body>\n\n\n  <!--\n    Facet name: unknown\n    Page type: main_only\n    Has global search: True\n    Has application tabs: True\n    Has side portlets: False\n\n    At least 5 queries/external actions issued in 0.06 seconds OOPS-c410218fca12b647b51d880c16fd0f89\n\n    Features: {\'js.yui_version\': None, \'visible_render_time\': None, \'hard_timeout\': None, \'app.mainsite_only.canonical_url\': None, \'app.maintenance_message\': None, \'baselayout.careers_link.disabled\': None}\n\n    r91d5f4c\n\n    -->\n\n</html>\n\n'
---


Error fetching 'github':
Traceback (most recent call last):
  File "PROJECT_ROOT/startriage/triage.py", line 176, in _await_and_print
    result: TriageResult = await task
                           ^^^^^^^^^^
  File "PROJECT_ROOT/startriage/sources/github/triage.py", line 256, in find
    results = await asyncio.gather(*tasks)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "PROJECT_ROOT/startriage/sources/github/triage.py", line 245, in _fetch_repo
    return await fetch_repo(
           ^^^^^^^^^^^^^^^^^
    ...<6 lines>...
    )
    ^
  File "PROJECT_ROOT/startriage/sources/github/finder.py", line 228, in fetch_repo
    issues, prs = await asyncio.gather(
                  ^^^^^^^^^^^^^^^^^^^^^
    ...<2 lines>...
    )
    ^
  File "PROJECT_ROOT/startriage/sources/github/finder.py", line 190, in _fetch_items
    data = await _graphql(session, query, variables)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "PROJECT_ROOT/startriage/sources/github/finder.py", line 119, in _graphql
    raise RuntimeError(f"GitHub GraphQL HTTP {resp.status}: {text[:200]}")
RuntimeError: GitHub GraphQL HTTP 403: {"message":"API rate limit exceeded for REDACTED_IP_ADDRESS. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)","documentation_url":"

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions