What's the philosophy of HtmlUnit when a response contains a header "Content-Type: application/octet-stream"

[qurikuduo]

Hi there,
Some URL have a response with header "Content-Type: application/octet-stream". Should I process it as an attachment?
After some digs, The Attachment only handle specific response which define in rfc-2183.
the :
attachmentHandler_.isAttachment(webResponse)
will return False when we have "application/octet-stream".
I found org.htmlunit.HttpWebConnection.downloadContent() will be called:
public static DownloadedContent downloadContent(final InputStream is, final int maxInMemory)
It will download the response content.
If I DON'T want HtmlUnit to download big content( e.g. https://dg.10000gd.tech:12348/shmfile/100 ), what should I do?
I want to block download action if a resource lager than 20MB to save on bandwidth.

Thanks a lot.

[rbri]

Maybe a simple solution is to set up your own WebConnectionWrapper and intercept the request url's. For the large ones don's call super and simply return a static response.

see https://www.htmlunit.org/faq.html#HowToModifyRequestOrResponse as starting point

[rbri]

will try to make a bit more detailed description ....

[qurikuduo]

Sounds like an option.

1. Specify my own WebConnectionWrapper.
2. Try to get content-length which defined in response Headers
3. If content-length not defined, try to implement my own HttpWebConnection implements WebConnection interface, then I will determine the response body is too large to be blocked in public static DownloadedContent downloadContent():
   When
   while( readCount = InputStream.read(buffer) !=0){ //... }
   Is it a solution?
   thx.

[qurikuduo]

After trying a few small tricks, I achieved the functionality I wanted.
Here is what I did:

1. Specify my own WebConnectionWrapper copied from HttpWebConnection and put it in package org.htmlunit : public class MyxxHttpWebConnection extends HttpWebConnection,
   Override public WebResponse getResponse(final WebRequest webRequest)
   and get content-length by read : httpResponse.getFirstHeader(ContentLength).getValue() ,
   determine if it is too large:
   if(contentLengthLong> maxContentLength){
   System.out.println("Content is too big. url="+webRequest.getUrl().toString()+" contentLength = " + contentLengthLong + ", maxContentLength = " + maxContentLength);
   httpMethod.abort();
   httpResponse.setEntity(null);
   }

2. Specify my own AttachmentHandler: public class MyxxAttachmentHandler implements AttachmentHandler
   @OverRide:
   public void handleAttachment(final Page page) {
   //not download attachment lager than 100KB
   if(page.getWebResponse().getContentLength() > maxAttachmentSize){
   System.out.println("Attachment is too big. url=" + page.getUrl()+" contentLength = " + page.getWebResponse().getContentLength() + ", maxAttachmentSize = " + maxAttachmentSize);
   try {
   page.getEnclosingWindow().getWebClient().getWebConnection().close();
   }catch(Exception e){
   logger.error("Error when close attachment download.", e);
   }
   finally {
   try {
   page.getWebResponse().cleanUp();//new AbstractPage(page.getWebResponse(),page.getEnclosingWindow())) ;
   page.getEnclosingWindow().setEnclosedPage(new HtmlPage(createWebResponse(new WebRequest(page.getUrl(),page.getWebResponse().getWebRequest().getHttpMethod()), "",
   page.getWebResponse().getContentType(), page.getWebResponse().getStatusCode(),page.getWebResponse().getStatusMessage()),page.getEnclosingWindow()));
   } catch (Exception e) {
   logger.error("Error when close attachment download.", e);
   }
   return;
   }
   }
   else {
   //if not response
   collectedAttachments_.add(new Attachment(page));
   }
   }

3. Create new instance before calling getPage(url):

webClient.setAttachmentHandler(new MyxxAttachmentHandler(attachmentList) );
new WebConnectionWrapper(webClient) {
public WebResponse getResponse(WebRequest request) throws IOException {
MyxxHttpWebConnection webConnection = new MyxxHttpWebConnection(webClient);
return webConnection.getResponse(request);
}
};
page=webClient.getPage(url)
if(attachmentList.size()>0){
//download attachment.
long contentLength = attachement.getPage().getWebResponse().getContentLength();
if(contentLength==0||(contentLength>MyxxAttachmentHandler.maxAttachmentSize)){
System.out.println("attachment too large, will not save to disk. contentLength = "+contentLength);
continue;
}
else{
//save attachment to file.
}
}

It is work for me now.

[rbri]

Hi @qurikuduo,

slowly i got an idea what you like to do.
I made some small changes and now i can do something like this.

@Test
public void contentBlocking() throws Exception {
    final byte[] content = new byte[] {77, 44};
    final List<NameValuePair> headers = new ArrayList<>();
    headers.add(new NameValuePair("Content-Encoding", "gzip"));
    headers.add(new NameValuePair(HttpHeader.CONTENT_LENGTH, String.valueOf(content.length)));

    final MockWebConnection conn = getMockWebConnection();
    conn.setResponse(URL_FIRST, content, 200, "OK", MimeType.APPLICATION_JSON, headers);

    startWebServer(getMockWebConnection());

    final WebClient client = getWebClient();
    client.setWebConnection(new HttpWebConnection(client) {
        @Override
        protected WebResponse downloadResponse(final HttpUriRequest httpMethod,
                final WebRequest webRequest, final HttpResponse httpResponse,
                final long startTime) {

            // check the header here if you like
            // call return super.downloadResponse() in case you are happy with the headers

            httpMethod.abort();

            // create empty response and mark as blocked for later
            final DownloadedContent downloaded = new DownloadedContent.InMemory(null);
            final long endTime = System.currentTimeMillis();
            final WebResponse response = makeWebResponse(httpResponse, webRequest, downloaded, endTime - startTime);
            response.markAsBlocked("test blocking");
            return response;
        }
    });

    final UnexpectedPage page = client.getPage(URL_FIRST);
    assertTrue(page.getWebResponse().wasBlocked());
    assertEquals("test blocking", page.getWebResponse().getBlockReason());
}

Will this help to simplify your code? do you need some other changes for your case?

[rbri]

@qurikuduo just made a new snapshot build - please try

3.4.0-SNAPSHOT

[rbri]

Have update the documentation a bit - https://www.htmlunit.org/details.html
Hope that helps.

[rbri]

Will close this, hope the changes and the docu are sufficient

[qurikuduo]

Than you very much.