net.sourceforge.htmlunit.corejs.javascript.EvaluatorException: Exception: java.lang.NullPointerException in HtmlUnit 2.43

[sowjanyaboddeti]

Getting the below exception while calling getPage for url: https://investors.alticeusa.com/investors/alticeusa/results-and-presentations/default.aspx using HtmlUnit 2.43.0 version but version 2.23 is not throwing the exceptions.
Exception in thread "main" net.sourceforge.htmlunit.corejs.javascript.EvaluatorException: Exception: java.lang.NullPointerException at net.sourceforge.htmlunit.corejs.javascript.Context.reportRuntimeError(Context.java:1040) at net.sourceforge.htmlunit.corejs.javascript.Context.reportRuntimeError(Context.java:1090) at com.gargoylesoftware.htmlunit.javascript.host.css.CSSStyleSheet.loadStylesheet(CSSStyleSheet.java:410) at com.gargoylesoftware.htmlunit.javascript.host.html.HTMLLinkElement.getSheet(HTMLLinkElement.java:143) at com.gargoylesoftware.htmlunit.html.HtmlLink$1.execute(HtmlLink.java:311) at com.gargoylesoftware.htmlunit.html.HtmlLink.onAllChildrenAddedToPage(HtmlLink.java:321) at com.gargoylesoftware.htmlunit.html.parser.neko.HtmlUnitNekoDOMBuilder.endElement(HtmlUnitNekoDOMBuilder.java:560) at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source) at com.gargoylesoftware.htmlunit.html.parser.neko.HtmlUnitNekoDOMBuilder.endElement(HtmlUnitNekoDOMBuilder.java:514) at org.apache.xerces.parsers.AbstractXMLDocumentParser.emptyElement(Unknown Source) at net.sourceforge.htmlunit.cyberneko.HTMLTagBalancer.startElement(HTMLTagBalancer.java:773) at net.sourceforge.htmlunit.cyberneko.HTMLTagBalancer.emptyElement(HTMLTagBalancer.java:820) at net.sourceforge.htmlunit.cyberneko.filters.DefaultFilter.emptyElement(DefaultFilter.java:145) at net.sourceforge.htmlunit.cyberneko.filters.NamespaceBinder.emptyElement(NamespaceBinder.java:310) at net.sourceforge.htmlunit.cyberneko.HTMLScanner$ContentScanner.scanStartElement(HTMLScanner.java:2769) at net.sourceforge.htmlunit.cyberneko.HTMLScanner$ContentScanner.scan(HTMLScanner.java:2117) at net.sourceforge.htmlunit.cyberneko.HTMLScanner.scanDocument(HTMLScanner.java:927) at net.sourceforge.htmlunit.cyberneko.HTMLConfiguration.parse(HTMLConfiguration.java:506) at net.sourceforge.htmlunit.cyberneko.HTMLConfiguration.parse(HTMLConfiguration.java:459) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at com.gargoylesoftware.htmlunit.html.parser.neko.HtmlUnitNekoDOMBuilder.parse(HtmlUnitNekoDOMBuilder.java:760) at com.gargoylesoftware.htmlunit.html.parser.neko.HtmlUnitNekoHtmlParser.parse(HtmlUnitNekoHtmlParser.java:208) at com.gargoylesoftware.htmlunit.DefaultPageCreator.createHtmlPage(DefaultPageCreator.java:283) at com.gargoylesoftware.htmlunit.DefaultPageCreator.createPage(DefaultPageCreator.java:163) at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:638) at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:451) at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:368) at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:520) at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:501) at com.factset.htmlunit.testapp.main(testapp.java:99)

[rbri]

Can you please add your WebClient setup and try with the latest version 2.44.

[sowjanyaboddeti]

WebClient :

WebClient webClient = new WebClient(BrowserVersion.getDefault());
webClient.setWebConnection(new MyWebConnection(webClient, args[0]));
webClient.setIncorrectnessListener(new MyIncorrectnessListener());
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.setJavaScriptTimeout(20000);
webClient.waitForBackgroundJavaScript(100000);

WebConnection Wrapper:

private static class MyWebConnection extends WebConnectionWrapper {
            String initialUrl = null; // url of the monitored page
            int responseCount = 0;
            int MAX_COUNT = 1000;
            
            public MyWebConnection(WebClient webClient, String initUrl) throws IllegalArgumentException {
                    super(webClient);
                    initialUrl = initUrl;
            } // override
            @Override
            public WebResponse getResponse(WebRequest request) throws IOException {
                    WebResponse response = null;
                    String url = request.getUrl().toString();
                    if (responseCount++ > MAX_COUNT) {
                            System.out.println("Total number of responses from this site exceed " + MAX_COUNT
                                            + ". It is possible this is an infinite loop. Initial Url:" + initialUrl);
                            throw new IOException("Total number of responses from this site exceed " + MAX_COUNT
                                            + ". It is possible this is an infinite loop. Initial Url:" + initialUrl);
                    }
                    try {
                            response = super.getResponse(request);
                            if (((response.getStatusCode() != 200) && !url.equalsIgnoreCase(initialUrl))
                                            && (response.getStatusCode() != 302) && (response.getStatusCode() != 301)) {
                                    System.out.println("create an empty web response in order to ignore exception due to this : "
                                                    + response.getStatusCode() + " " + response.getStatusMessage() + " " + url);
                                    response = null;
                            }
                            String origCharset = (response.getContentCharset()).toString();
                            if (origCharset.equalsIgnoreCase("unicode")) {
                                    String newContent = response.getContentAsString(Charset.forName("utf-8")).replaceAll("(?i)charset\\s*=\\s*unicode",
                                                    "charset=utf-8");
                                    response = new StringWebResponse(newContent, Charset.forName("utf-8"), request.getUrl());

                                    /*String origCharset = response.getContentCharset();
                            if (origCharset.equalsIgnoreCase("unicode")) {
                                    String newContent = response.getContentAsString("utf-8").replaceAll("(?i)charset\\s*=\\s*unicode",
                                                    "charset=utf-8");
                                    response = new StringWebResponse(newContent, "utf-8", request.getUrl());*/
                                    System.out.println("Page Id: change charset " + origCharset + " ==> " + "utf-8");
                            }
                    } catch (IOException ex) {
                            if (url.equalsIgnoreCase(initialUrl))
                                    throw ex;
                            System.out.println("create an empty web response in order to ignore this exception: " + ex.getMessage());
                            response = null;
                    }
                    return response;
            }
    }

    private static class MyIncorrectnessListener implements IncorrectnessListener {
    @Override
    public void notify(String arg0, Object arg1) {
    }

Getting the same exception on 2.44 version as well.

[rbri]

did a first quick test

• what should i use for args[0] here webClient.setWebConnection(new MyWebConnection(webClient, args[0]));

Maybe the reason for the NPE is your code - at two places you are setting the response to null (response = null;)

[sowjanyaboddeti]

I am sorry, I left out that detail. It is the URL string that goes as args[0].

I am setting the response to NULL only incase of IOException or incase the response status code IS NOT 200. Should that be a problem?

Also, I am confused how webClient.getPage() and WebConnection.getResponse() are wired internally?

[rbri]

for me it looks like the problem is in your code

if (((response.getStatusCode() != 200) && !url.equalsIgnoreCase(initialUrl))
    && (response.getStatusCode() != 302) && (response.getStatusCode() != 301)) {
    System.out.println("create an empty web response in order to ignore exception due to this : "
            + response.getStatusCode() + " " + response.getStatusMessage() + " " + url);
    response = null;
}
String origCharset = (response.getContentCharset()).toString();

If you code sets the response to null, the next line after the if block will produce the NPE.

[sowjanyaboddeti]

Working now. Thanks for your help.