Error 503 - Service Unavailable

[PJR-slo]

• [Y ] API

When attempting to submit a VAT return, Error 503 - Service Unavailable is returned precisely 1 minute after the POST command, both in the live and sandbox environments.

All other functions (retrieve obligations, liabilities, payments and VAT returns) work as expected.

This was first reported to me by clients on Friday, 1st November 2019. They have previously used the same software to submit returns.

The most recent instance of a VAT return being successfully submitted with this software is on the 4th October 2019, so it would appear that something has changed since then.

When submitting a VAT return, the software first retrieves the obligations and then submits the return in the same session. Below is an example of the sandbox communications:

2019-11-05T09:52:52Z Getting obligations: https://test-api.service.hmrc.gov.uk/organisations/vat/196274233/obligations?from=2017-04-01&status=O&to=2017-06-30
2019-11-05T09:52:52Z GET /organisations/vat/196274233/obligations?from=2017-04-01&status=O&to=2017-06-30 HTTP/1.1
2019-11-05T09:52:53Z {"obligations":[{"due":"2017-05-07","end":"2017-03-31","periodKey":"18A1","received":"2017-05-06","start":"2017-01-01","status":"F"},{"due":"2017-08-07","end":"2017-06-30","periodKey":"18A2","start":"2017-04-01","status":"O"}]}

2019-11-05T09:52:53Z Posting VAT Return: https://test-api.service.hmrc.gov.uk/organisations/vat/196274233/returns
2019-11-05T09:52:53Z POST /organisations/vat/196274233/returns HTTP/1.1
2019-11-05T09:52:53Z POST data: {"periodKey":"18A2","vatDueSales":10456.95,"vatDueAcquisitions":20.00,"netVatDue":10406.95,"vatReclaimedCurrPeriod":70.00,"totalVatDue":10476.95,"totalValueSalesExVAT":52284.00,"totalValuePurchasesExVAT":250.00,"totalValueGoodsSuppliedExVAT":200.00,"totalAcquisitionsExVAT":100.00,"finalised":true}
2019-11-05T09:53:53Z HTTP/1.1 503 Service Unavailable

I have tried to delete and refresh the tokens in both live and sandbox environments, but this made no difference.

At all times of testing, the service availability has been "Full service available"

As stated above, the software has previously been able to submit VAT returns in both live and sandbox environments from all clients sites and my offices.

Unfortunately I have a deadline to get this fixed, so an early response would be greatly appreciated.

Thanks in advance

Peter

[mtd-api-team]

Hi @PJR-slo 👋

We can see your requests and have raised the issue with the team responsible for access to the HMRC API platform. We don't have an answer yet, but are looking into the problems you are having.

We will keep you updated.

[nigelrainer]

We also have clients getting service unavailable and internal server errors on obligations requests.

We have raised it with SDST.

[PJR-slo]

@nigelrainer This could be something to do with invalid or malformed headers.

We understand that a 503 (server unavailable) error can be generated because the VAT-API is transforming a different error (i.e. 415) into a 503.

Error 415 (which I cannot find being documented anywhere) is actually "Unsupported Media Type" and can refer to the "Content-Type=" header not being present or malformed.

Of course, it can also refer to a load of other issues.

Apparently the server validation rules have been significantly tightened up very recently which caused our software to fail and yet I never received any notifications that these changes were made.

Peter

[PJR-slo]

We have now solved our issue, but there are faults that remain in the VAT-API, the documentation and also in the procedures for making changes to the VAT-API and therefore I will leave this fault report OPEN for now.

Whilst submitting a VAT return (and only submitting), our software contained a malformed header:

Content-Type: application/json; charset=utf-8, application/json

This header worked fine in both sandbox and live environments up to about the 24th October 2019, when apparently significant changes were made to the VAT-API.

Unfortunately, we were not notified of these changes and we became aware of them on the 4th November 2019, when clients started to report that they were unable to submit VAT returns. Further investigations showed that Error 503 (service unavailable) was being returned by the VAT-API.

With a deadline to submit the returns of 7th November, we only had 3 days to find a solution to the problem - in the belief that this was a server error.

Eventually we received a reply from SDSTeam stating that there was a fault in the VAT-API whereby the server was timing out and returning the wrong error code. The code that should have been returned was 415. I have been unable to find this error code documented.

In conclusion I would ask that the following points be considered for inclusion into the procedures for the VAT-API development team:

1. Ensure that all documentation is up-to-date, especially with regard to error codes
2. Do not release untested (and faulty) software to the "live" environment
3. Whenever changes are made to the VAT-API:
   a. Release to the Sandbox only
   b. Immediately notify all developers to allow for proper testing
   c. Only release to the "live" environment at least one month after notification to developers
4. In circumstances such as this, ensure that helpful responses to developers are made in a timely manner.

Thank you for your assistance in this matter.

Peter

[nigelrainer]

We have checked our headers and they are all fine, so I do not believe this matter is limited to the headers raised above - so this definitely needs to be kept open and monitored. PJR-slo's points above pretty much covers everything HMRC have done to frustrate the developer community on the VAT API since day one.

[PJR-slo]

@nigelrainer if you have a look at the changes made to v1.151.0, you will see they are quite extensive with many of them adding more validation.

It is quite likely that other incorrectly handled errors are also timing out and returning a 503. Unfortunately, the only way to find out the actual error is to contact SDSTeam, who will be able to see what is happening at their end.

Peter

[mtd-api-team]

@PJR-slo the changes that have been made recently to the VAT API codebase are library and framework updates. They have no bearing on the existing behaviour of the VAT API, as was proved by extensive testing before being deployed into Sandbox and then Live the following week. The changelog is the best place to check when changes are deployed into the respective environments.

The issues you are having appear to be with connectivity to HMRCs API gateway - something that SDST are in the best position to expedite. This team is not best placed to advise or support on this matter, as we are one of many API producer teams on the platform.

[hmrc-api-team]

Hello, we are closing this issue due to age. If you still have any persisting issues please feel free to open a new issue.