What are people's experience upgrading to Oxalis 5.0.0? (or 5.0.1)

[OysteinLq]

Hi, some weeks back Oxalis 5.0.0 was released. I'm curious if anyone has made the upgrade yet, and what their experience with it has been like.

We are planning to do the upgade fairly soon, so I can post a bit about it when that happens in case anyone else is interested. As we're running Oxalis in a somewhat offbrand¹ manner, some of our experience will likely differ from most people's.
Our initial tests do seem to indicate that it will be a simple upgrade.

_{1: We've wrapped Oxalis in some of our own code to create an interface we prefer, and we're running it in Wildfly}

[OysteinLq]

In case anyone sees this and haven't upgraded, I can inform you that this was a completely painless process for us at Logiq-- aside from having to update some imports in our own codebase, it was pretty much just a simple drop-in of the dependencies. (And we were happy to have an official AS4 release via Maven as well!)

So thanks to everyone involved with this release :)

[OysteinLq]

Spoke too soon!
We're experiencing a Nullpointer from the depths of Oxalis on a certain pair of recipients, therefore we've reverted back to the previous Oxalis release. Except for the SMP-lookup of those two recipients, we've had no issues.

See issue #513

[aaron-kumar]

Added comment in #513 but good to receive any lead/logs from you to reproduce this issue. I mentioned the constraint of Production AP certificate and availability of these endpoints in SML only (not SMK) , so any additional information will provide helping hand.

[OysteinLq]

Since I'm being chatty about our situation here, I can give some data on our experience so far:
We've kept the Oxalis 5.0.0 Inbound running in production, and see no problems with it.
In our test environment, we're running the patched version of SMP & Outbound, as described in issue %513, mentioned above. We're seeing no problems with it, though obviously we're not doing a lot of tests of the actual outbound sending.

We did send a few thousand files (Sorry, I haven't an exact number) through the upgraded Outbound module in production before turning it off, and generally found it to work perfectly fine. As mentioned in the aforementions Issue, of the thousands of lookups we performed, there were only two particular identifiers that led to a Nullpointer. That was still a serious enough issue for us to choose to revert it. When we reverted just the SMP, we found that the Outbound module failed with the exact same issue, since a new SMP lookup is performed as part of that process.

We probably won't go into production with 5.x.x for the SMP and Outbound until the Vefa patch is part of the official release.

[aaron-kumar]

Sorry, I have bit difficulty in understanding your comment. You mentioned "patched version of SMP". Are you hosting participant "0192:917537534" and "9908:943574537" in your SMP?

[OysteinLq]

Hi, sorry, that's imprecise language on my part. By SMP in this case I meant our SMP Lookup Service (which we run as a sort of microservice in our architecture).

We do not host either of these participants.

[aaron-kumar]

Thanks @OysteinLq . That clarify it.

[aaron-kumar]

Oxalis 5.0.1 released (https://github.com/OxalisCommunity/oxalis/releases/tag/v5.0.1) . Please test and confirm

[jannewaren]

Hello! Thanks a lot for trying it out and sharing your experience here, this is very valuable!

Personally I have not tried out Oxalis 5.0.0 yet but we are in the process of doing the upgrade. So, it sounds like there is a need for a new Oxalis release where this fixed vefa-peppol library is taken into use. Hopefully we can have that release out soon.

[OysteinLq]

We're testing 5.0.1 now, and will probably give it a go in production sometime next week. As mentioned on the Issue, we can confirm that our previously reported problem seems to be resolved now. Will report back later.

[OysteinLq]

Welp, we've given 5.0.1 a shot in production for 5 hours, and decided to revert back to 4.1.2.
Unfortunately, we don't have a great amount of useful data to share as yet.

First: the Inbound module seems to work perfectly well.
SMP lookups have given us problems, however, both directly and via the Outbound module.

The plus: we stopped seeing the connection timeouts on SMP lookups that we've had since sometime last year.
The negative: We got a set of new, related errors, which our systems handled less well. They did seem to be related to socket timeouts or failed connections

We've seen 36 instances of this problem in our logs during those 5 hours. I don't have the exact count of the number of lookups in that period, but would estimate it to about 10,000. There does seem to be a bit of clustering of failed lookups, often with 2 or more lookups failing at the same time.

The errors were something ala:

Unable to fetch 'http://B-0656bcfdf25c9800465437e8820236be.iso6523-actorid-upis.edelivery.tech.ec.europa.eu/iso6523-actorid-upis%3A%3A0192%3A820956532/services/busdox-docid-qns%3A%3Aurn%3Aoasis%3Anames%3Aspecification%3Aubl%3Aschema%3Axsd%3AInvoice-2%3A%3AInvoice%23%23urn%3Acen.eu%3Aen16931%3A2017%23compliant%23urn%3Afdc%3Apeppol.eu%3A2017%3Apoacc%3Abilling%3A3.0%3A%3A2.1

(Note: this is just a random example. The problem does not seem to be related to the particular participant or their access point)

Looking through the Vefa codebase, this seems to usually originate from a SocketTimeout and SocketException in UrlFetcher and BasicApacheFetcher. Unfortunately we didn't have a full stacktrace to see more than just the error message for these cases.

Mind, we've run this with the same settings we used with earlier Oxalis. We've not yet experimented with non-default values in the new oxalis.http.pool configuration values.

We will be looking into this more, of course, hoping to reproduce it in our test environments, but thought someone might be interested in hearing about our experience thus far.

[jannewaren]

Hi! Thanks for the detailed report once again 👍

Can you share when was that 5 hour period exactly? We have not yet upgraded to Oxalis 5 but we are seeing this problem constantly. So far the assumption is that the SMP in question just has some trouble coping with the number of requests they get, and later we retry and all is good.

If your problematic timeframe aligns with our experience that is further proof maybe that it's not perhaps about Oxalis 5 but just the SMP being unresponsive. I will attach the hourly error graph (for this exact error) here if you want to see if we had those same errors at the same time.

I will try to take a look if the problem happens only for certain SMP or many of them and get back to this.

[jannewaren]

For example today has been a good day, the total traffic has been pretty similiar to yesterday but yesterday we got a lot of these errors, today only a few.

Or more specifically actually, during the day (EEST) there were not much errors yesterday either. But during the evening (around 18-23) there were a lot. And it does not correspond at least to our high volume times, if anything the opposite. But can't know about the traffic from other service providers of course.

[OysteinLq]

Thanks Janne, I cannot tell from the graphs, but here's some exact timestamps for times of failure while running 5.0.1 (these denote when it fails, not when the initial request started)
All from May 19th.
10:26:54,163
10:26:54,163
10:45:28,743
10:45:28,743
10:56:14,039
10:56:14,039
10:56:14,039
10:56:14,039
11:04:33,366
11:24:26,599
11:55:18,795
11:57:07,611
11:59:53,720
12:04:50,276
12:23:59,537
13:15:12,171

We do still get the daily I/O exception
(java.net.SocketException) caught when processing request to {}->http://smp.difi.no:80: Connection timed out (Read failed)
Which is one of the things we were hoping upgrading to 5.0.x would fix.
This timeout problem first started appearing for us in September 2020, without us having really made any change to our system. We've been in dialog with Digdir to see if there might be issues with Elma, but so far they haven't seen anything on their end, though they could confirm that we weren't alone in struggling with this.

[aaron-kumar]

Thanks @OysteinLq for sharing details. We are aware of SMP lookup problems and these problems are reported by couple of Oxalis users as early as 2020. Oxalis 5.0.0/5.0.1 did not meant to address SMP problem (there were other business need as well) but this issue is top of our list and we are prioritizing it accordingly (and we may probably will come back with some of recommendations as how you can get rid of those problems).
Please feel free to contribute via PR and sharing detailed analysis like you did. Thanks !!!

[carher]

We have been running 5.0.0 for a few weeks now. No problems yet. We have fairly little traffic currently, though.

However, our motivation for upgrading was to hopefully fix this issue: #494
No luck, though, hasn't made a difference for us.