Why are 403 response codes sent while ROR plugin isn't initiated?

[usev6]

After upgrading to ROR 1.37.0 I saw some issues with clients that got back a 403 response code for requests to a freshly restarted elasticsearch node.
At that point the ROR plugins wasn't fully initiated yet (which was pretty clear from the exception and the log entries from Elasticsearch).

If I'm not mistaken this change was introduced with #745.

I'm curious what motivated that change. Could you maybe disclose what the mentioned issue RORDEV-530 was about?

As a background: The Elasticsearch RESTClient library treats 503 errors as a reason to retry agains the next elasticsearch node -- which doesn't happen with the 403 response codes.

• https://github.com/elastic/elasticsearch/blob/3f6d460d01/client/rest/src/main/java/org/elasticsearch/client/RestClient.java#L347-L351
• https://github.com/elastic/elasticsearch/blob/3f6d460d01/client/rest/src/main/java/org/elasticsearch/client/RestClient.java#L566-L574

[coutoPL]

Hi @usev6

I've checked our Jira and the motivation was pretty prosaic - our users were often misled by huge ROR stack trace produced by ES when ROR returned 5xx HTTP response (you know, when you want to return 5xx error you have to prepare a proper exception and pass it through ES internals).

We thought about it a little bit and we agreed that from the perspective of security, we're just forbidding requests to ES until we are not sure if the request is authorized. When ROR is starting (and it has to start asynchronously) the incoming requests cannot be authorized because ROR is not ready yet, so ROR just rejects them. ROR forbidden response contains the due_to field where we add the cause.

I understand that we lost the retrying mechanism provided by the RestClient. TBH, we were not aware.

I don't think we want to go back to the previous solution as a default. But maybe we can provide an option to override the default behavior for power users (403 by default and 503 when you know what you are doing).

Would it work for you?

[usev6]

Hi @coutoPL

thanks for your prompt reply and your explanation. The solution you suggested (an option to override the default behaviour) sounds very good to me. I'm pretty sure it would solve the problem I'm seeing here.

[coutoPL]

ok, I created a Jira for this (RORDEV-643). We will try to implement it when we are less busy.
I will let you know about it here, in this thread.

[coutoPL]

@usev6 we have it implemented. It will be released with ROR 1.48.0. If you want to test it I can send you a pre-build (give me a hint about the ES version you use) or you can build it yourself (develop branch).

ref: #884

[coutoPL]

It was released with ROR 1.48.0 (see docs)