Potential memory leak

[lklonowski]

We are using HtmlUnitDriver to run many tests in parallel, usually about 40.
With HtmlUnitDriver 2.23.2, a maximum heap of 6 GB was enough for that purpose. Now that we switched to 2.26, we are getting OutOfMemoryErrors.
I reduced parallelity down to 25 test threads and went up to 10 GB max heap. Still, heap is always full, garbage collection running non-stop but no free memory can be allocated.
When I'm looking at the heap, the class which uses most of the heap ist net.sourceforge.htmlunit.corejs.javascript.ScriptableObject with several million instances.

Strange thing is, we have JavaScript disabled in the WebClient, so I don't even see why all these instances are necessary.

[asashour]

Can you provide more details?

A simple complete case is always helpful.

[lklonowski]

The simplest thing I could come up with which is not project related is a google search. I bet it can be replaced by simple requests on any other web page.
I started the attached program with VM arguments '-Xmx10g'. After 5 minutes, the 7GB of old gen space were full. After 7 minutes, the garbage collector activity didn't drop under 90% any more.

[lklonowski]

Attachment didn't work, here is the code:

import java.util.Random;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;

import org.openqa.selenium.By;
import org.openqa.selenium.Keys;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.htmlunit.HtmlUnitDriver;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;

public class Main {

	public static void main(final String[] args) throws InterruptedException {
		int numberOfThreads = 25;
		ExecutorService executor = Executors.newFixedThreadPool(numberOfThreads);
		for (int i = 0; i < numberOfThreads; i++) {
			executor.execute(new TestRunner(i));
		}
		executor.shutdown();
		executor.awaitTermination(10, TimeUnit.MINUTES);
	}

	private static class TestRunner implements Runnable {

		private static final Random RANDOM = new Random();

		private static final char FIRST = ' ';

		private static final char LAST = '~';

		private final int id;

		public TestRunner(final int id) {
			this.id = id;
		}

		@Override
		public void run() {
			HtmlUnitDriver driver = new HtmlUnitDriver(false);
			for (int i = 0; i < 1000; i++) {
				driver.get("https://www.google.de/");
				String searchString = getRandomSearchString();
				WebElement input = new WebDriverWait(driver, 1).until(ExpectedConditions.elementToBeClickable(By.name("q")));
				input.sendKeys(getRandomSearchString() + Keys.RETURN);
				int resultcount = driver.findElements(By.xpath("//div[@id='rso']/div/div/div[@class='g']")).size();
				System.out.println("Thread " + id + " returned " + resultcount + " results for searchstring " + searchString);
			}
		}

		private String getRandomSearchString() {
			// use between 10 and 20 chars
			int numberOfChars = RANDOM.nextInt(11) + 10;
			StringBuilder sb = new StringBuilder(10);
			for (int i = 0; i < numberOfChars; i++) {
				// use chars from space ' ' up to tilde '~'
				char c = (char) (RANDOM.nextInt(LAST - FIRST + 1) + FIRST);
				sb.append(c);
			}
			return sb.toString();
		}

	}

}

[lklonowski]

Is there any further information I can provide to track this issue down?

[rbri]

Hope this is fixed. Please try with the latest snapshot.

[lklonowski]

I tested the 2.29-SNAPSHOT of htmlunit driver and didn't run into any memory problems any more.
I encountered some performance issues which I couldn't track down and may be caused by the fact that I couldn't find an perfectly working Selenium version for this snapshot. But this very issue here can be regarded as closed.

[rbri]

Thanks for the feedback. Feel free to open a new issue if you think there is a new performance problem.