Support POST as HTTP method

[allanw]

This particular API I'm dealing with (aXcelerate) requires that some requests be sent as POSTs. Could the request type be set in the config?

[daigotanaka]

@allanw Can you answer these 2 questions?

Q1. Can I confirm that you are not trying to write anything to aXcelerate via tap-rest-api? Just reading the data out?
Q2. Can you point me to the document that describes the API endpoint you are trying to invoke?

I'm asking this because it's unusual for an API to use POST method while it is not creating or modifying the record.
If you are trying to use tap-rest-api for modifying the data on aXcelerate, tap-rest-api is not a tool to do so.

[allanw]

Hi @daigotanaka,

1. This is just reading data out - no data is being written in this case
2. The particular endpoint in question is called /report/run (https://admin.axcelerate.com.au/apidocs/) All it does is runs a report which I have saved and returns the data as JSON

I realise that this is an unusual scenario - I'm not actually sure why their API requires this to be a POST method.

[daigotanaka]

Hi @allanw

Thanks for the clarification and reference.

I'm not quite sure if this fits to tap-rest-api's intended cases for these 2 reasons (I'm not familiar with aXcelerate's, so I may be wrong):

1. From reading the doc, it seems /report/run actually triggers aXcelerate's report running process. It would update aXcelerate's stored values instead of just reading out. (Therefore, the POST method instead of GET)
2. I looked at the example response object. This does not look like what singer.io typically handles. The record is usually a stream of JSON records either with newline separated format or a list (array) within a JSON object that wraps around it. Each record should have a See timestamp_key, datetime_key, index_key section.

For #2, The example I see on doc is:

{
    "DATA": [
        {
            "Qualification Name": "Certificate I in Business",
            "Organisation": "VM Learning / aXcelerate",
            "Class Descriptor": "# W OUT DATES",
            "Trainer Full Name": "",
            "Actual Completion Date": "",
            "Full Name": "Nathan Gordon",
            "Commencement Date": "2015-07-17",
            "USI": "",
            "Qualification Code": "BSB10112",
            "Enrolment Date": "2015-07-17",
            "Expected Completion Date": "2015-07-17"
        }
    ],
    "ERRORMSG": "",
    "REPORTNAME": "Qualification Enrolments and Outcomes",
    "PATH": "",
    "SUCCESS": true,
    "COUNT": 1,
    "CONTENTTYPE": "application/json",
    "FILTERS": [
        {
            "VALUE2": "",
            "NAME": "enrolments.enrolstatus",
            "VALUE": "comp"
        },
        {
            "VALUE2": "",
            "NAME": "enrolments.dateenrolled",
            "VALUE": "01/01/2015"
        },
        {
            "VALUE2": "",
            "NAME": "contacts.givenname",
            "VALUE": "Nathan"
        },
        {
            "VALUE2": "",
            "NAME": "contacts.surname",
            "VALUE": "Gordon"
        }
    ],
    "DESCRIPTION": "JSON/Collection"
}

...which seems like a one-page report than a list of records.

It is technically possible to modify tap-rest-api to call this endpoint with POST method and pull the data out. But I'm hesitant in supporting this in this tap because POST call typically requires data as payload. Providing logic to prepare the payload depends on the applications and the use cases. There isn't a good way of authoring that other than developing custom code.

It seems your objective is to call this API periodically to pull the report and load it on a destination data store. Wouldn't it be simpler to just call it with Python's requests library and store the result?

I'm curious to know why you think singer.io or tap-rest-api is suited for this use case. Let me know what you think. More than happy to help what you need to achieve whether you use tap-rest-api or not.

[allanw]

Hi @daigotanaka, thanks for the detailed reply!

1. I see what you mean here and you're right, this API endpoint actually runs a report. However, I'm not sure if it does actually update any stored values. If it does, then a POST would make sense, but I'm also new to this API so I could be wrong too.
2. In that example, there is only one result in the DATA array. In my case, this will in fact return an array with lots of objects in it, one for each row of data in the report. This is the only thing I care about retrieving from the API, and I can get this working by setting record_list_level: "DATA" in the config file.

To get this working in my test environment, I did actually modify tap-rest_api code a little bit to be able to do a POST request.

You make a good point that this could be done using the requests library and then storing the result, but I was keen to stay within the Singer tap/target ecosystem as this is how I'm extracting other sources of data on a regular schedule, and ideally I'm trying to avoid writing too much bespoke code.

Interested to hear any more feedback you have. I appreciate your help and understand why you'd be hesitant about adding POST as an additional method without a good use case for it.