REST2 API support

[sebix]

I think I finally found documentation for it:
https://metacpan.org/pod/RT::Extension::REST2

AS it is JSON-based it will be easier to implement and less error-prone I guess.

[sebix]

https://github.com/bestpractical/rt-extension-rest2/blob/master/README

[lelutin]

Hi there, I'm very interested in having REST2 support since this version of the API has fixed a couple of annoyances and makes for a better basis for mechanized interaction with RT.

One of the major annoyances is that REST 1.0 sends out values in its response that correspond to the translated strings that would end up getting shown to a user on the web interface. For example dates for tickets are localized which makes it annoying to parse and unreliable if we must expect to parse dates for all possible localized patterns.

The REST2 API should be somewhat easy to implement (famous last words) and I'm interested in working on this. However from what I can see from the codebase we'd need to decouple a bunch of functions.

Do you think that implementing support for REST2 as a different (new) file entirely would make sense?

[sebix]

Hi there, I'm very interested in having REST2 support since this version of the API has fixed a couple of annoyances and makes for a better basis for mechanized interaction with RT.

Cool!

Do you think that implementing support for REST2 as a different (new) file entirely would make sense?

Honestly, I have no idea how to do this properly. Do you know if the provided API call set is approximately the same and do they have different names and parameters?

[lelutin]

Honestly, I have no idea how to do this properly. Do you know if the provided API call set is approximately the same and do they have different names and parameters?

It's very similar yes. You can see a list of API endpoints here:

https://github.com/bestpractical/rt-extension-rest2/blob/master/README#L371

So far the endpoints I've looked at were the same except with 2.0 in the URL and they're sending out json instead of some weird format. So response parsing will need to be completely overhauled. Currently, response parsing for api 1.0 is all intertwined in the code.. so that's why I was wondering if a new file would be less efforts.

Also for login, it's possible to use user/login as with 1.0 but you can also pair the API up with the other extension https://metacpan.org/release/RT-Authen-Token to use a sort of "per-application" token instead of the user credentials directly. So we'll have to implement this as well

[lelutin]

I've tried to put down some ideas to get a rough plan of what needs to be done for this issue. If there's something that seems useless or extraneous or missing in there, let me know.

1. testing

• where should these tests reside?
  • creating a tests/ directory would make things clearer
• the current test file looks more like acceptance tests that need a live
  RT instance
• if we end up keeping some code in common between API 1.0 and 2.0 then
  we'll probably want to create some kind of unit testing that mocks
  external dependencies

2. Create a client for REST2 in a separate file

• login
• logout
• system information (through the /rt endpoint)
  • testing that login works requires us to call an endpoint to see if we
    get the correct http code in response. We could use the /rt
    endpoint for this test, which could then always be used to populate
    system information locally. then we'd avoid additional calls to get
    the RT version and the list of installed RT extensions.
• operations on different endpoints
  • idea: data that we get back from RT through the API should probably
    "smell" the same between both api versions. If we create object
    classes for data from the different endpoints, we could make it
    easier for users to interact with data.
  • CRUD for each, but for some types of objets there might be additional
    actions. for example tickets can be commented on and replied to
  • tickets
  • transactions
  • attachments
  • queues
  • assets
  • catalogs
  • users
    • group membership
  • groups
  • custom fields
    • some actions relate to other types of objects: catalog, class, queue
      they seem to only relate to searching for custom fields related to
      these objects so this should probably translate to an action on
      each of the three types for searching for custom fields, which
      would return a list of custom field objects.
  • custom roles

[e3rd]

(I'll enthusiastically support anything that works and receives no serious objection from the community, looking forward to the PR.)

[ghost]

* where should these tests reside?
  
  * creating a `tests/` directory would make things clearer

+1

* the current test file looks more like acceptance tests that need a live
  RT instance

* if we end up keeping some code in common between API 1.0 and 2.0 then
  we'll probably want to create some kind of unit testing that mocks
  external dependencies

OTOH testing against a real RT instance ensures that we are compatible.

[vnznznz]

@lelutin I'm also interested in working on this. Whats you status? RT 5.0 with included REST2.0 will be released "soonish"

[lelutin]

@vinzBad I frankly haven't had the time to start on this yet. but the responses here are giving me more motivation :)

really nice to learn that the API v2.0 will be built-in to the next major release 👍

So then, like you say, it's really important to build support for this version of the API sooner rather than later.

Maybe the places to start would be twofold:

1. to talk about the kind of class structure we'd like to have? REST1.0 was implemented in one class that returns arrays and hashes, but for 2.0 I was thinking of going more towards splitting things out into smaller blocks representing data. e.g.

• The main class is used as the "connector" to login and send API calls
• Tickets have a class of their own
• Queues too
• same of other primitive data that the API sends out to the users

do you think I'm overthinking it?

2. We need to settle on the test suite and how we mock data for unit tests

I'm not super comfortable on that front since I don't do all that much python development on the day-to-day basis, so I'm all ears for suggestions.

[vnznznz]

I have several thoughts about this.

• I would like to provide a port of the current rt class that uses REST2 as backend, because this fixes some problem current python-rt couldn't solve with REST1 and it is the accepted python rt interface
  • would be most of use now, because existing users wouldn't need much migration effort to use REST2
• I also like the idea of refactoring the existing api to a bit more object oriented design but this needs to solve some pain points the current api has (like returning 'ticket_id/$TICKETID' instead of $TICKETID when creating ticket) or provide some cool features (like supporting pagination for search results)
  • feedback from python-rt community would be great for this

RE test-suite and mock data

• for REST2 some JSON based mock is enough I think
• don't have have a preference, maybe something more modern like pytest or nose2
• with the RT docker images I think we don't need a lot of unit tests, an investment in better acceptance/integration tests would have more impact imho

[ghost]

• I also like the idea of refactoring the existing api to a bit more object oriented design but this needs to solve some pain points the current api has (like returning 'ticket_id/$TICKETID' instead of $TICKETID when creating ticket) or provide some cool features (like supporting pagination for search results)

I like the idea of creating a "new" refactored class which not only supports RT's new REST API but also has a better design.

[lelutin]

* I would like to provide a port of the current rt class that uses REST2 as backend, because this fixes some problem current python-rt couldn't solve with REST1 and it is **the** accepted python rt interface
  
  * would be most of use now, because existing users wouldn't need much migration effort to use REST2

We could mimick the inputs to functions currently available for the REST1 client, but since REST2 responses differ, we'd have to put in extra effort to be fully compatible. So I feel like making it fully transparent to users who switch between using REST1 and REST2 will be a challenge, except if we change the API for the REST1 class as well.

[vnznznz]

You're right. A new REST2 client shold come first. A port für the REST1 interface maybe second. I just learned on the bestpractical forums that there already is a python REST2 implementation: https://gitlab.com/catalyst-cloud/rt-client Maybe there is some inspiration to be found.

[rgrueebler]

I would be very interested in the REST2 support. Specially, in the token authentication.

[sim0nx]

FWIW I have been playing around a bit with the new API. Unfortunately outputs have changed and the documentation is not as good/complete as it looks at first glance IMO.
Also a number of endpoints do not exist yet (e.g. for managing links) and others are cumbersome to use (like fetching attachments).

We'll see how this new API evolves.

[lelutin]

You're right. A new REST2 client shold come first. A port für the REST1 interface maybe second. I just learned on the bestpractical forums that there already is a python REST2 implementation: https://gitlab.com/catalyst-cloud/rt-client Maybe there is some inspiration to be found.

ah thanks for pointing that one out. I've tried it out a bit and while the HTTP client works and lets one request some data (I was able to mostly migrate my simple scripts for work), it is rather incomplete since there's no simplified way to interact with a couple different objects. also its testing is rather inexistant (only one test for a query building function).

the REST2 API is itself also rather disappointingly still quite buggy and unfinished :( (for example I can't find any way to create links between tickets or merge tickets) but it is mostly there and we could have a lot of functionality up and running with it.

so, first things first: what test suite should we chose?

What do ppl participating in this issue say about using pytest?

[lelutin]

What do ppl participating in this issue say about using pytest?

fwiw the current test file test_rt.py uses unittest, so we could also continue on with this.

[ghost]

so, first things first: what test suite should we chose?

What do ppl participating in this issue say about using pytest?

As long as it works and the library is not exotic, I don't mind. Both unittest and pytest are fine for me.

[sim0nx]

so, first things first: what test suite should we chose?
What do ppl participating in this issue say about using pytest?

As long as it works and the library is not exotic, I don't mind. Both unittest and pytest are fine for me.

I would prefer pytest :-)

[lelutin]

so, first things first: what test suite should we chose?
What do ppl participating in this issue say about using pytest?

As long as it works and the library is not exotic, I don't mind. Both unittest and pytest are fine for me.

I would prefer pytest :-)

I'll switch over to pytest for v2 tests in my next work burst. the v1 tests will stay as they are unless someone ends up converting them later. I'll start by just using pytest as a library for the test files and see if running those inside nose2 causes a problem or not. This way, we can keep CI as it is setup currently.

on another subject: I've just tried using catalyst-cloud's rt-client lib with an auth token without supplying user and password and while it was successful at grabbing the auth page (which is unnecessary with a token) subsequent requests failed because the token wasn't saved and reused in request headers.

[arekm]

Note that REST 2.0 API in RT::Extension::REST2 is incomplete when compared to REST 2.0 API in rt 5.0 itself - https://rt.cpan.org/Public/Bug/Display.html?id=134781

[sebix]

Thank you @sim0nx for all your work on this feature! 🎉

[sim0nx]

Thank you @sim0nx for all your work on this feature! tada

👍