RFE: Support bugzilla REST API

[crobinso]

For bugzilla 5 and later, the XMLRPC API is deprecated and won't be receiving new features, and eventually removed (though I suspect not for a while). So we need to add support for the REST API to python-bugzilla soon

[xhe123]

When the REST API will be supported by python-bugzilla? Thanks!

[djmitche]

GothenburgBitFactory/bugwarrior#697 shows Mozilla dropping support for the XMLRPC APi. Will support be added here, soon, or should we plan to call the REST API directly in bugwarrior?

[djmitche]

I took a look at what it would mean to add REST support to this library -- XMLRPC is pretty deeply ingrained, and I suspect it would be difficult or impossible to retain the same Python and CLI APIs with a --use-rest-api option.

How would you see this being done?

[crobinso]

I was hoping to get to this over xmas break or early january, but no guarantees. Here's how I see it happening:

• find all the XMLRPC API usage. git grep "\._proxy" will cover most of that
• add a bugzilla/backendxmlrpc.py with a BugzillaBackendXMLRPC class. for starters, this can take the self._proxy instance as an init argument.
• initialize it in BugzillaBase: self._backend = BugzillaBackendXMLRPC(self._proxy)
• add wrapper calls for every API we call that is flagged with the grep command. For example, self._proxy.Bug.update could become:

class BugzillaBackendXMLRPC(object):
    ...
    def bug_update(self, args):
        return self._xmlrpc.Bug.update(args)

• Convert all the API usage to use wrappers like that. Verify test suite still works

That's the mechanical part that should be API transparent. From there you can split out a base class _BackendBase that defines a common API, start implementing a BugzillaBackendRest, and continuing to make sure the API passes as new APIs are implemented. Something like that. I'm happy to take patches for any of those steps. Doing it as incrementally as possible will make it easier for review and testing

[djmitche]

Thanks! That sounds like a good plan!

[djmitche]

Hm, the second step has turned out to be a lot more complicated: REST doesn't support login, cookies, etc., so a great deal of the functionality involving the rcfile, config options, etc., need to change. The RHBugzilla hack is a bit confusing in there, too.

I'm not sure I'll be able to make this work, at least not in a way that would make any sense in a review or have any chance of actually working against bugzilla's other that Mozilla or in modalities (such as CLI) other than the one I use.

[crobinso]

Thanks for attempting it. If you get something even partially working it's useful to share. If the REST API means that existing functionality no longer works, so be it; we will just have to live with it. So even if you can get just one auth method working (API key I presume), and hack that into place, but then with that get most other API usage to work in a relatively clean way, that's a big step in the right direction. We could also start a side branch work work-in-progress will happen

[crobinso]

Re: side branch, even if you get something working that full on comments out code or changes the default to be REST based, that can be a step for further exploring. So don't worry about making it play nice with the existing code in the early stages

[djmitche]

Any chance we can drop the RHBugzilla support? That's .. quite a hack.

[djmitche]

In particular, I'm getting errors like these:

Traceback (most recent call last):
  File "/home/dustin/p/python-bugzilla/tests/test_query.py", line 109, in testSubComponent
    self._sub_component_out)
  File "/home/dustin/p/python-bugzilla/tests/test_query.py", line 39, in clicomm
    assert out == q
AssertionError: assert {'component': ['lvm2', 'kernel'],\n 'include_fields': ['assigned_to', 'id', 'status', 'summary'],\n 'sub_components': ['Command-line tools (RHEL5)']} == {'component': ['lvm2', 'kernel'],\n 'sub_components': ['Command-line tools (RHEL5)']}
  Common items:
  {'component': ['lvm2', 'kernel'],
   'sub_components': ['Command-line tools (RHEL5)']}
  Left contains 1 more item:
  {'include_fields': ['assigned_to', 'id', 'status', 'summary']}
  Full diff:
    {
     'component': ['lvm2',
                   'kernel'],
  -  'include_fields': ['assigned_to',
  -                     'id',
  -                     'status',
  -                     'summary'],
     'sub_components': ['Command-line tools (RHEL5)'],
    }

and I think it's because the rh detection is failing, but I can't quite figure out how it used to work with the tests (it depends on url being set, but url isn't set..)

[djmitche]

Here's my work so far

[crobinso]

We probably can't drop it entirely from the API, like fully removing the class. But if we can fold the behavior into the base class and just keep RHBugzilla as an alias for Bugzilla, that would be fine. If for a first pass RHBugzilla is totally busted, that's fine with me, I'll need to dig more into it all to give useful feedback.

I'll try to check your code in the latter half of this week. Thanks again!

[crobinso]

I've pushed a complete REST implementation upstream. The only public compatibility issues that I know of are:

• update_tags API seems to have been dropped on the bugzilla side. it was marked as experimental so this isn't entirely unexpected
• redhat ExternalBugs extension doesn't seem to have a REST entrypoint
• redhat bugzilla doesn't have /component PUT/update in their instance, though it's in the upstream docs
• Some API pieces I didn't test yet that require admin access to bugzilla: user create and user edit, component create and component edit.

There's some small heuristics like, if we see /rest in the URL, assume the user is requesting REST, so it can be tested from the command line that way. Otherwise we still default to XMLRPC

To force use of REST via the API, use Bugzilla(url=FOO, force_rest=True, ...)

Any testing will be much appreciated!