Proxy Not Renewing Token

[juicetiger1]

I am having an issue where I can get a token and the application functions properly. But after the 1 hour expiry elapses the token is not renewed by the resource proxy.

We have been able to determine the issue (with the help of ESRI support) in the code at line 213 in proxy.jsp, where the code is checking for specific error codes.

This string used to check the error code is not being decoded and appears in a "wing-ding" format and the error codes are not found. If the error code is not found the renew token function is not triggered.

I have not been able to reproduce this error on other systems that use the resource proxy and it is only occurring with this application.

Here is a (modified) copy of the proxy.config we are using:

<?xml version="1.0" encoding="utf-8" ?>
<ProxyConfig allowedReferers="*"
                logFile="e:\WEB-INF\classes\proxy_log.log"
                logLevel="FINEST"
                mustMatch="true">
  <serverUrls>
        <serverUrl url="https://<URL>/arcgis/rest/services"
        matchAll="true"
             tokenServiceUri="https://<URL>/portal/sharing/rest/generateToken/"
               username="username"  
               password="password"
        />
        
  </serverUrls>
</ProxyConfig>

[AmrEldib]

I've been working with @juicetiger1 on this issue.

Here are my testing notes if it helps address the issue:
Issue: Resource Proxy (Java) fails to Renew Token after it Expires

Notes:

• Client is running resource proxy 1.1.2
  https://github.com/Esri/resource-proxy/releases/tag/v1.1.2
• The proxy is configured to allow access to allow access to a secured Geocode service via tokenServiceUri, Username, Password combination

<serverUrl url="https://citymaps.peterborough.ca/arcgis/rest/services/PtboLocatorLocateApp/GeocodeServer"
matchAll="true"
tokenServiceUri="https://citymaps.peterborough.ca/portal/sharing/rest/generateToken"
username="XXXXX"
password="XXXXX"
/>

• ArcGIS Server is federated with Portal. Both Server and Portal are at 10.6.1

• The resource proxy is hosted on Tomcat web server.

• The proxy is able to successfully generate a token and use it without a problem.

• After an hour passes, the token expires, and the resource proxy fails to recognize that the token is expired, and doesn't attempt to generate a new token.

• The proxy log shows no errors

• We inspected the Java code responsible for renewing the token

• After adding a line to print log of the response from Server that includes any error messages, we found that the proxy prints some incoherent text instead of a proper string response.

• Following the logic of the code in line 212, this string won't match any of the conditions of the If statement and won't be able to recognize the error returned from the server/portal
  https://github.com/Esri/resource-proxy/blob/master/Java/proxy.jsp#L212

• The issue seems limited to this one installation of the resource proxy

• Client has other applications setup with their own resource proxy (also, Java on Tomcat) and they all work fine with similar configuration.

• I've attempted to reproduce the issue, but couldn't. The proxy works fine.

• We're looking for help identifying what's causing the resource proxy to behave this way.

[cpore]

@juicetiger1, @AmrEldib,

I've run into what may be the same issue. If so, I've managed to fix/workaround it. I was getting the same "wing-ding" response body when one particular service sent along an authentication error that looked like this when written to the log:

I actually found two issues when handling the authentication error, but the root cause is that the authentication error is gzip encoded. Now, I didn't dig deep enough to determine if this encoding issue is with the proxy code, something else in the call stack (like a tomcat, or connection issue), or if the problem was with the server itself sending an ill-formed response (it would be nice to know why other responses aren't gzipped, or if they are, why they get properly decoded and this one doesn't). But either way I was able to put in a work-around to gzip decode the response if other means didn't catch the authentication error, like so:

    //proxy gets the response back from server
    private boolean fetchAndPassBackToClient(HttpURLConnection con, HttpServletResponse clientResponse, boolean ignoreAuthenticationErrors) throws IOException{
        if (con!=null){
            //copy the response content to the response to the client
            InputStream byteStream;
            if (con.getResponseCode() >= 400 && con.getErrorStream() != null){
                if (ignoreAuthenticationErrors && (con.getResponseCode() == 498 || con.getResponseCode() == 499)){
                    log.info("Authentication error found in response code.");
                    return true;
                }
                byteStream = con.getErrorStream();
            }else{
                byteStream = con.getInputStream();
            }

            ByteArrayOutputStream buffer = new ByteArrayOutputStream();
            int length = 5000;
            byte[] bytes = new byte[length];
            int bytesRead;

            while ((bytesRead = byteStream.read(bytes, 0, length)) > 0) {
                buffer.write(bytes, 0, bytesRead);
            }

            //if the content of the HttpURLConnection contains error message, it means the token expired, so let proxy try again
            //only check small responses (< 1KB). It's probably expensive to convert and parse large responses every time.
            log.info("RESPONSE BUFFER SIZE: " + buffer.size());
            if(buffer.size() <= 1024 && !ignoreAuthenticationErrors){
                String strResponse = buffer.toString();
                log.info("RESPONSE: " + strResponse);
                if (hasResponseBodyAuthenticationError(strResponse)) {
                    log.info("Authentication error found in response body.");
                    return true;
                }

                //Do a gzip decode and check those results as well
                if("gzip".equals(con.getHeaderField("Content-Encoding"))) {
                    GZIPInputStream gzin = new GZIPInputStream(new ByteArrayInputStream(buffer.toByteArray()));
                    ByteArrayOutputStream outputBytes = new ByteArrayOutputStream();
                    int read = 0;
                    byte gzBytes[] = new byte[1024];

                    while ((read = gzin.read(gzBytes, 0, gzBytes.length)) > 0) {
                        outputBytes.write(gzBytes, 0, read);
                    }

                    String strGzipResponse = outputBytes.toString();
                    log.info("CONTAINS GZIP HEADER. UN-GZIPPED RESPONSE: " + strGzipResponse);
                    if (hasResponseBodyAuthenticationError(strGzipResponse)) {
                        log.info("Authentication error found in gzipped response body.");
                        return true;
                    }
                }
            }

            copyHeadersToResponse(con, clientResponse);

            clientResponse.setStatus(con.getResponseCode());

            byte[] byteResponse = buffer.toByteArray();
            OutputStream ostream = clientResponse.getOutputStream();
            ostream.write(byteResponse);
        }
        return false;
    }

Since I'm unable to determine exactly what conditions cause the decoding issue, I just check all responses for errors that aren't caught via the first check by gzip decoding them, and optimize to only check small-ish responses to save some cycles.

You also may notice that I moved the code that sets the response headers to occur after the authentication error check. I found that when it's before the check, it adds duplicate headers that cause issues at the client since that section of code would be called twice if an authentication error occurred.

Finally, you may notice I removed all the flush() and close() calls, as those are unnecessary in Java 8, so be sure to leave those in if using something lower.

Anywho, I hope this helps you (or anybody) out. Apologies, I can't submit a pull request for a fix because I've refactored this project into a Spring Boot project to make it more easily deployable in a Kubernetes cluster.

[cpore]

On another, slightly related note, I've also found that some error responses have been missed because they contain a space before and after the colon and should be checked, like so:

private boolean hasResponseBodyAuthenticationError(String responseBody){
        return responseBody.contains("error")
               && (responseBody.contains("\"code\": 498")
                   || responseBody.contains("\"code\": 499")
                   || responseBody.contains("\"code\":498")
                   || responseBody.contains("\"code\":499")
                   || responseBody.contains("\"code\" : 498") //add this check
                   || responseBody.contains("\"code\" : 499")); // and this check
    }

[juicetiger1]

Are you able to send the copyHeadersToResponse function as well?

Thanks

[cpore]

@juicetiger1

Are you able to send the copyHeadersToResponse function as well?

Thanks

I've made no changes here, other than re-arranging the code. I simply extracted all of the header-setting code at the top of the method into its own method and moved it below the authentication checking code.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you need additional assistance please contact Esri Technical Support. Thank you for your contributions.

[github-actions]

This issue has been automatically closed due to inactivity. If you need additional assistance please contact Esri Technical Support.

[ranga-tolapi]

Are you able to send the copyHeadersToResponse function as well?

Thanks

private void copyHeadersToResponse(HttpURLConnection con, HttpServletResponse clientResponse){
	Map<String, List<String>> headerFields = con.getHeaderFields();
	Set<String> headerFieldsSet = headerFields.keySet();

	//copy the response header to the response to the client
	for (String headerFieldKey : headerFieldsSet){
		//prevent request for partial content
		if (headerFieldKey != null && headerFieldKey.toLowerCase().equals("accept-ranges")){
			continue;
		}

		List<String> headerFieldValue = headerFields.get(headerFieldKey);
		StringBuilder sb = new StringBuilder();
		for (String value : headerFieldValue) {
			// Reset the content-type for OGC WMS - issue #367
			// Note: this might not be what everyone expects, but it helps some users
			// TODO: make this configurable
			if (headerFieldKey != null && headerFieldKey.toLowerCase().equals("content-type")){
				if (value != null && value.toLowerCase().contains("application/vnd.ogc.wms_xml")){
					_log(Level.FINE, "Adjusting Content-Type for WMS OGC: " + value);
					value = "text/xml";
				}
			}

			// remove Transfer-Encoding/chunked to the client
			// StackOverflow http://stackoverflow.com/questions/31452074/how-to-proxy-http-requests-in-spring-mvc
			if (headerFieldKey != null && headerFieldKey.toLowerCase().equals("transfer-encoding")) {
				if (value != null && value.toLowerCase().equals("chunked")) {
					continue;
				}
			}

			sb.append(value);
			sb.append("");
		}
		if (headerFieldKey != null){
			clientResponse.addHeader(headerFieldKey, DataValidUtil.removeCRLF(sb.toString()));
		}
	}
}