Algo fetch: HTTP errors aren't displayed in the logs

[AurelienGasser]

HTTP errors occurring during the algo fetch aren't reported in the logs. This makes it hard to understand what is going on.

For instance, in this issue, the logs only error show hash doesn't match [...] which is unhelpful: the real source of the problem is the HTTP 403 error which occurred prior to the hash computation.

[samlesu]

It looks like it is similar to the issue #217. The cause of the failure is different, but it's the same ambiguous error message.

[AurelienGasser]

After more investigation, the case I was looking at was not a HTTP 403 error. Despite the appearances, the server returns a 200.

For context, this is what the worker pod receives when it tries to fetch e.g. susbtra-backend.node-1.com/algo/.... but its DNS resolution points to the public internet, and not the k8s cluster.

Maybe another approach would be to look at the response content type. Thoughts @Kelvin-M

[AurelienGasser]

I'm going to close this for now as there's no easy way to fix it.

We should keep in mind that errors like hash doesn't match ... are usually due to DNS misconfiguration. The solution is make sure that the address substra-backend.node-{1-N}.com resolves to the cluster IP, and not to the public internet. This could be part of a troubleshooting guide? cc @natct10

[natct10]

I'm going to close this for now as there's no easy way to fix it.

We should keep in mind that errors like hash doesn't match ... are usually due to DNS misconfiguration. The solution is make sure that the address substra-backend.node-{1-N}.com resolves to the cluster IP, and not to the public internet. This could be part of a troubleshooting guide? cc @natct10

You're right! I will add this to the troubleshooting section, thank you for pointing this one out @AurelienGasser