Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

500 response on submission API #887

Open
charles-f opened this issue Feb 4, 2022 · 26 comments
Open

500 response on submission API #887

charles-f opened this issue Feb 4, 2022 · 26 comments

Comments

@charles-f
Copy link

charles-f commented Feb 4, 2022 •
edited
Loading

API

Describe the bug
Hi - we're seeing a few 500 responses on the submission API, but the submission is actually going through as the obligation itself becomes fulfilled with a received date that matches the submission date. However since the API for the submission returned 500 - we don't have the receipt and correlation id for that submission.

To Reproduce
This is intermittent - but it's simply submitting through the API

Expected behavior
201 response along with the response headers and body

Additional context
In the response body - I've noted it returns a HTML error response - which includes an ID - maybe this error id means something to you?

I'm trying to find a pattern but it just seems random.

Live environment
@DDCTLS-DEV-TEAM
Copy link

Good Morning @charles-f
Thank you for raising this one. Do you have the error ID? and We can take a look and see if we can de-bug this one for you. The intermittent part is the difficult part but we will try and find a solution.
Kind Regards

@charles-f
Copy link
Author

charles-f commented Feb 8, 2022 •
edited
Loading

Good morning @DDCTLS-DEV-TEAM

the response on the API received was this. Which is an error ID - This exception has been logged with id 7mgib5j5i

<title>Error</title> <style type="text/css"> html, body, pre { margin: 0; padding: 0; font-family: Monaco, 'Lucida Console', monospace; background: #ECECEC; } h1 { margin: 0; background: #A31012; padding: 20px 45px; color: #fff; text-shadow: 1px 1px 1px rgba(0,0,0,.3); border-bottom: 1px solid #690000; font-size: 28px; } p#detail { margin: 0; padding: 15px 45px; background: #F5A0A0; border-top: 4px solid #D36D6D; color: #730000; text-shadow: 1px 1px 1px rgba(255,255,255,.3); font-size: 14px; border-bottom: 1px solid #BA7A7A; } </style>

Oops, an error occurred

This exception has been logged with id 7mgib5j5i.

There are more but this is an one of them. Can you please let me know how we can get the submission receipt and correlation IDs for that submission.

Can you expand the VAT returns API endpoint to include this information, then at least we have a way to retrieve that even if there is a failure in that original submission?

@DDCTLS-DEV-TEAM
Copy link

We have done some digging into the code as well as Kibana Logs. There are three stages that we have observed:

Stage 1: The actual submission at “January 31st 2022, 17:42:00.695” that resulted in Timeout Exception from the Downstream service (DES). Unfortunately we could not log any meaningful information at this stage as its Timeout exception.

Stage 2: The subsequent tries made in the next couple of minutes resulted in error: “The VAT return was already submitted for the given period”.
This positively indicates that the submission made in Stage-1 was indeed successful.

Stage 3: The subsequent attempts to retrieveObligations are all returning the error: “The remote endpoint has indicated that no data can be found”. (This might also means a positive response as the data needed to return is no longer present).

We can return or log ‘submission receipt’ and ‘correlation ID’ only when the Downstream returns success response. Its not possible to return them from VAT-API to the caller unless the Downstream gives them back to VAT-API.
To understand exactly with what’s happening with the VRN in question (we have the VRN) and to get the submission receipt and correlation IDs for that submission, we are going to write to DES Support, but it will take a while to get the response back from them. We can update when we have response from DES Support.

@aeverest
Copy link

Hi, any update please? Thanks.

@DDCTLS-DEV-TEAM
Copy link

Hi, we have provided all the information requested by DES Live Support team. They came back and asked if the VAT Regs we provided are right. We have confirmed they were correct. We are still waiting in the response from them.

@hmrc hmrc deleted a comment from DDCTLS-DEV-TEAM Feb 24, 2022
@aeverest
Copy link

Hi, any update please? Thanks.

@DDCTLS-DEV-TEAM
Copy link

DDCTLS-DEV-TEAM commented Mar 24, 2022 via email

@charles-f
Copy link
Author

Hi, Any further updates - it's now been a while and we're keen to get some constructive feedback. And we're approaching a new submission window where I think it will happen again.

We've seen this on pretty much every quarterly filing window - 500 http code responses on submission - though I don't know if they are all the same.

In 2022 alone - I've seen 14 submission failures with 500 http code responses. In December 2021 - another 11. List goes on.

Thanks,
Charles

@charles-f
Copy link
Author

Hi, Any further updates - it's now been a while and we're keen to get some constructive feedback. And we're approaching a new submission window where I think it will happen again.

We've seen this on pretty much every quarterly filing window - 500 http code responses on submission - though I don't know if they are all the same.

In 2022 alone - I've seen 14 submission failures with 500 http code responses. In December 2021 - another 11. List goes on.

Thanks, Charles

With the new deadline now passed, we did see 12 more occurrences of 500 responses on submissions.

  • 8 on 3rd May
  • 3 on 4th May
  • 1 on 6th May

that's 12 attempts across 5 different VRNs. Any update on this situation because it's only happening more and more.

@hmrc hmrc deleted a comment from DDCTLS-DEV-TEAM May 11, 2022
@charles-f
Copy link
Author

Hi any update - this still continues to happen.

@DDCTLS-DEV-TEAM
Copy link

DDCTLS-DEV-TEAM commented Jul 6, 2022 via email

@karolskrobot
Copy link

Can we have an update please?

We have the same issue occurring for a few hundred submissions each month - 500 responses followed by inconsistent replies from vat return and obligations endpoints. We reported this via the SDS team with detailed data as requested, and we've been waiting for any response since 10 June.

As additional note, returning he HTML response is a bug in itself. No content-type header is set so it defaults to application/json. We would expect the body to be JSON like any other error response.

@charles-f
Copy link
Author

Hi this is still happening a lot.

In the July August filing period - we've had 103 of these 500 responses on the submissions.

Surely this is a serious enough bug for you to prioritise?

@DDCTLS-DEV-TEAM
Copy link

Hello @charles-f

For us as a team it is our priority and to be honest, we are feeling your frustration too. We have had to escalate this and we now have a meeting booked for tomorrow with the team that owns the data (Unfortunately that's not us).

We are hoping we will have something tangible tomorrow for you or at least an update on what the plan is.
We apologize for the inconvenience this is causing. It's just a bit out of our hands until we finally got the meeting booked.

Thank you again for your patience and we'll update you tomorrow.

@charles-f
Copy link
Author

thanks @DDCTLS-DEV-TEAM appreciate the update.

@charles-f
Copy link
Author

hi @DDCTLS-DEV-TEAM can you please let us know the update from today's meeting please

thanks

@DDCTLS-DEV-TEAM
Copy link

@charles-f Good morning,

The outcome of the meeting was that the backend systems have acknowledged there is an issue.
They think that the backend system that they hit, doesn't update in time. They pull the period key from the backend which doesn't give a response in time, and they send us the error. However, the backend then sends them the correct response it just doesn't come in time.

So if you have a recent example over the last 30 days and if you can send the data to the SDS team we can send this on.
Sorry, they need a recent error.

Thank you for your help in this one, hopefully, we are getting somewhere with it. Unfortunately, it's something we cant sort ourselves on the vat-api team.

@aeverest
Copy link

Great to hear that there is recognition of a source of this, looking forward to a solution :-)

@LH200
Copy link

LH200 commented Aug 12, 2022 •
edited
Loading

Hello,

Is there an ETA on when a fix will be applied for this?

We have had reports of this issue on the latest batch of returns.

A quick fix to ensure that the submission is not posted to HMRC would be preferable in the short term.

At the moment our system will state the submission hasn't worked because we expect a JSON response and not html:

<title>Error</title> <style type="text/css"> html, body, pre { margin: 0; padding: 0; font-family: Monaco, 'Lucida Console', monospace; background: #ECECEC; } h1 { margin: 0; background: #A31012; padding: 20px 45px; color: #fff; text-shadow: 1px 1px 1px rgba(0,0,0,.3); border-bottom: 1px solid #690000; font-size: 28px; } p#detail { margin: 0; padding: 15px 45px; background: #F5A0A0; border-top: 4px solid #D36D6D; color: #730000; text-shadow: 1px 1px 1px rgba(255,255,255,.3); font-size: 14px; border-bottom: 1px solid #BA7A7A; } </style>

Oops, an error occurred

This exception has been logged with id EXAMPLE.

@charles-f
Copy link
Author

@LH200 thanks we have the exact same issue, and if you even attempt to resubmit then you get the rejection for duplicate submission also. Agreed I would prefer if the submission in fact doesn't happen as a result of this 500. And therefore a resubmission would be allowed.

This has now affected 121 of submissions in my product since June.

We're having to explain to users that it's an issue with HMRC's API and not our product.

Here is my latest example @DDCTLS-DEV-TEAM

<title>Error</title> <style type="text/css"> html, body, pre { margin: 0; padding: 0; font-family: Monaco, 'Lucida Console', monospace; background: #ECECEC; } h1 { margin: 0; background: #A31012; padding: 20px 45px; color: #fff; text-shadow: 1px 1px 1px rgba(0,0,0,.3); border-bottom: 1px solid #690000; font-size: 28px; } p#detail { margin: 0; padding: 15px 45px; background: #F5A0A0; border-top: 4px solid #D36D6D; color: #730000; text-shadow: 1px 1px 1px rgba(255,255,255,.3); font-size: 14px; border-bottom: 1px solid #BA7A7A; } </style>

Oops, an error occurred

This exception has been logged with id 7oi9c2i38.

@DDCTLS-DEV-TEAM
Copy link

@charles-f thank you for the update can you please send the VRN you used for that submission please to the SDS team and link in this Github issue they will forward it to us then thank you. Hopefully, the other teams then will have something to follow on with.

@LH200 unfortunately there isnt an ETA on this yet as the downstream services are still discussing the issue. The vat-api service and this team are basically a proxy to the backend services (different teams).
So far the investigations are leading to the fact that we as in vat-api are timing out waiting on a response from the backend (One team) we hit, which then goes off to another backend (another team) and that one doesn't respond in time. Is the current theory. So we need to confirm the full issue before a fix can go in place.

We also have an issue where the backend service isn't available at all, we have a fix going through the pipeline to stop the HTML being sent and the error is sent in JSON. So we are hoping once that goes in you won't see any HTML anymore and the JSON mite gives you more to go off in terms of letting your users know.

Thank you all for your understanding on this one, we are actively investigating this and trying to get the right people in the room.

@charles-f
Copy link
Author

We're still getting 500 errors on the response API, I've provided my team member who's in contact with Alex Anikeev about this issue - with the VRN for the follow up.

@DDCTLS-DEV-TEAM
Copy link

Good Morning @charles-f
Thank you for the update and thank you for pushing things on your side. We are doing the same. We have had an increase in this issue and we are sending example VRNs every week to try and push this through. As you can imagine the downstream teams are very busy but I'm hoping this can be solved sooner rather than later. We will keep you updated as soon as we get anything.

In the meantime, if you can keep emailing in your VRNs when you're getting the issues that will also help and we apologize for the inconvenience this is causing.

Kind Regards
DDCT Team

@DDCTLS-DEV-TEAM
Copy link

Good afternoon @charles-f
There's some confusion around the VRNs provided by your teammate. Could you provide the example VRNs to the SDS team please so we can proceed with the investigation with certainty? Submission timestamps would also be helpful if possible.

Kind regards
DDCT Team

@charles-f
Copy link
Author

@DDCTLS-DEV-TEAM what is your process for those affected by this who does not have a receipt id as a result.

Where the obligation has been fulfilled and users want a receipt acknowledging submission has been made.

We have customers asking about this bug, and though it maybe partially resolved in that you no longer show the html error anymore (what I see from my company's product), there are still some 500 responses on the submission (with json response) and many more 503s and some 504s than before.

My assessment of this would be that the 'fix' has only moved the issue to another response code (503s) and where 500s still happen (they now return json).

Can I suggest that if you do not get a response in time, then you reject the submission - this would allow the user to resubmit at least and not keep causing these issues.

I look forward to hearing your answer about what is the process for what to do about no receipt id.

@DDCTLS-DEV-TEAM
Copy link

Hello @charles-f

Thank you for getting in touch.
We have been investigating this since we last spoke and we have had some clarity of what is going on down stream.

So we (VAT-API) are hitting the downstream to submit. This has all gone through ok. Our downstream then go off further to update the user's data in the backends.

We (VAT-API) are throwing timeouts because our downstream doesn't respond in time. However, they have pushed that data onto the backends and it is in the process of updating/ submitting when we throw the timeout. Then after a certain amount of time (we are still in talks with the backends to understand this) our downstream then get a positive result but by then obviously, we have timed out.

Now we have upped our timeouts a little to try and reduce this problem. (as you can imagine this isn't a permanent fix and doesn't solve it for everyone) so we are trying to understand why these timesout are happening and what we can do about this. The VAT-API doesnt create the receipt id, that is given to us from a successful response. If the VAT-API is getting errors from the downstreams there is no way of us getting you that receipt id from inside the VAT-API service (at the moment). Unfortunately, at this time we dont know the ins and outs of how this is made and whether we could retrieve it at a later date, so say if you submitted the same thing, we could return something more useful like you have done this already here is the receipt id. We are trying to get to the bottom of this because that would help out alot of people like yourselves.

We will keep you updated as we go.

Kind Regards

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@karolskrobot @aeverest @charles-f @DDCTLS-DEV-TEAM @LH200