Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

Bug 792636 - Add some automated leak detection #598

Closed
wants to merge 13 commits into from

3 participants

@Mossop
Owner

I'd like to get some feedback on this @ochameau. It adds some very basic leak detection that displays if it thinks something has leaked at the end of the test run. It works using memory reporters. Before running tests it makes a list of all the JS compartments and windows that are in memory. After the test run it checks to see if anything new is there (ignoring a blacklist of chrome stuff). If there is it logs it.

Currently though it is not very reliable. Sometimes it will find a bunch of leaks, sometimes it won't. I ran it for every test individually and in the cases where it was reliably claiming a leak I always found a real cause for it. It now only reliably claims a leak for the private browsing module, and I think that is a real leak in platform code.

To try to make it more reliable I tried having it delay and re-check for up to a second, running GC each time, but still no joy. Do you have any ideas what might be going on?

The next step of this would probably be to also run the leak check after each test file, or even each individual test in the file to make it easier to narrow down the problems but that is pointless unless it gets more reliable.

@ochameau
Owner

Currently though it is not very reliable. Sometimes it will find a bunch of leaks, sometimes it won't. I ran it for every test individually and in the cases where it was reliably claiming a leak I always found a real cause for it. It now only reliably claims a leak for the private browsing module, and I think that is a real leak in platform code.

To try to make it more reliable I tried having it delay and re-check for up to a second, running GC each time, but still no joy. Do you have any ideas what might be going on?

Wouldn't it because the leaks are intermittent? I can easily imagine two cases: we leak only when a race condition happens (or the other way around ;)). And the platform code do stuff only on some runs, or only after xx seconds after firefox startup so that we will see more leaks on cfx testall just because it takes more time to execute.

Otherwise, I think that something simplier should already be reliable:

foceGC();
setTimeout(function () {
  forceGC();
  setTimeout(function () {
    checkLeaks();
  });
});

I would be interested so see cases where we need more forceGC.
Then, there is the nsIDOMWindowUtils method, I've always wondered if it was usefull to call it in addition to Cu.forceGC:
https://developer.mozilla.org/en-US/docs/XPCOM_Interface_Reference/nsIDOMWindowUtils#garbageCollect
Might be worth giving it a try?

Finally, I think that such feature can already be usefull if optional.

@Mossop Mossop Merge branch 'master' into check-memory
Conflicts:
	test/test-addon-installer.js
	test/test-content-symbiont.js
a2bf516
@erikvold

@Mossop I think we should cherry-pick the leak fixes that you've made at least.

lib/sdk/deprecated/unit-test-finder.js
@@ -11,6 +11,8 @@ module.metadata = {
const file = require("../io/file");
const memory = require('./memory');
const suites = require('@test/options').allTestModules;
+const { Loader } = require('../test/loader');
+const { ensure } = require('../system/unload');

can this line be removed?

@Mossop Owner
Mossop added a note

Yeah looks like it is left over from a previous iteration

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@erikvold

can we make this a optional feature for now and land it?

@Mossop
Owner

I've already cherry-picked the leak fixes and landed them elsewhere. I have been intending to turn this into an optional feature and get it landed, just haven't found the time yet. If you feel like taking it then by all means!

Mossop added some commits
@Mossop Mossop Merge branch 'master' into check-memory
Conflicts:
	lib/sdk/deprecated/unit-test-finder.js
	lib/sdk/test/harness.js
	lib/sdk/test/loader.js
	test/test-addon-installer.js
	test/test-page-mod.js
50e5e66
@Mossop Mossop Make memory checking optional. 4111415
@Mossop
Owner

I've updated this branch to master and made memory checking optional

@ochameau
Owner
error: reddit-panel: An exception occurred.
TypeError: panel is undefined
resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.path/sdk/
panel/utils.js 91
Traceback (most recent call last):
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/timers.js", line 31, in notify
    callback.apply(null, args);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/test/harness.js", line 226, in cleanup
    loader.unload();
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 41, in unloadWrapper
    originalDestructor.call(obj, reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/test/loader.js", line 31, in wrapper<.unload
    unload(loader, reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/toolkit/loader.js", line 379, in unload
    notifyObservers(subject, 'sdk:loader:destroy', reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/events.js", line 62, in
    data: data
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 77, in onunload
    unload(reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 56, in unload
    observers.forEach(function(observer) {
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 58, in
    observer(reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 67, in
    unloaders.slice().forEach(function(unloadWrapper) {
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 68, in
    unloadWrapper(reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 41, in unloadWrapper
    originalDestructor.call(obj, reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/test/loader.js", line 31, in
    unload(loader, reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/toolkit/loader.js", line 379, in unload
    notifyObservers(subject, 'sdk:loader:destroy', reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/events.js", line 62, in
    data: data
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 77, in onunload
    unload(reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 56, in unload
    observers.forEach(function(observer) {
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 58, in
    observer(reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 67, in
    unloaders.slice().forEach(function(unloadWrapper) {
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 68, in
    unloadWrapper(reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 41, in unloadWrapper
    originalDestructor.call(obj, reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/widget.js", line 338, in destroy
    this.panel.destroy();
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/core/disposable.js", line 70, in destroy
    this.dispose();
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/panel.js", line 170, in dispose
    this.hide();
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/panel.js", line 229, in hide
    domPanel.close(viewFor(this));
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/panel/utils.js", line 91, in close
    return panel.hidePopup && panel.hidePopup();
info: reddit-panel: [JavaScript Error: "reddit-panel: An exception occurred.
TypeError: panel is undefined
resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.path/sdk/
panel/utils.js 91
Traceback (most recent call last):
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/timers.js", line 31, in notify
    callback.apply(null, args);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/test/harness.js", line 226, in cleanup
    loader.unload();
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 41, in unloadWrapper
    originalDestructor.call(obj, reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/test/loader.js", line 31, in wrapper<.unload
    unload(loader, reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/toolkit/loader.js", line 379, in unload
    notifyObservers(subject, 'sdk:loader:destroy', reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/events.js", line 62, in
    data: data
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 77, in onunload
    unload(reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 56, in unload
    observers.forEach(function(observer) {
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 58, in
    observer(reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 67, in
    unloaders.slice().forEach(function(unloadWrapper) {
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 68, in
    unloadWrapper(reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 41, in unloadWrapper
    originalDestructor.call(obj, reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/test/loader.js", line 31, in
    unload(loader, reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/toolkit/loader.js", line 379, in unload
    notifyObservers(subject, 'sdk:loader:destroy', reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/events.js", line 62, in
    data: data
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 77, in onunload
    unload(reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 56, in unload
    observers.forEach(function(observer) {
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 58, in
    observer(reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 67, in
    unloaders.slice().forEach(function(unloadWrapper) {
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 68, in
    unloadWrapper(reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/system/unload.js", line 41, in unloadWrapper
    originalDestructor.call(obj, reason);
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/widget.js", line 338, in destroy
    this.panel.destroy();
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/core/disposable.js", line 70, in destroy
    this.dispose();
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/panel.js", line 170, in dispose
    this.hide();
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/panel.js", line 229, in hide
    domPanel.close(viewFor(this));
  File "resource://extensions.modules.anonid0-reddit-panel-at-jetpack.commonjs.p
ath/sdk/panel/utils.js", line 91, in close
    return panel.hidePopup && panel.hidePopup();
"]

r+ with this exception being fixed and commits squashed before merging.
(This exception happens when running cfx test -o in reddit-panel.)

Otherwise, I'm wondering if the memory check done in mochitest-browser can be done after each test without slowing down tests massively? (I'm excepting it to go through the whole CC graph...)
http://mxr.mozilla.org/mozilla-central/source/testing/mochitest/browser-test.js#321
We can identify any kind of leaks thanks to the CCAnalyzer helper. Here it is used to look for GlobalWindow, but we can track other objects as well, like Sandboxes...

@Mossop
Owner

Ok the exception is because my patch makes us properly unload the loader used for tests. This ends up calling panel.destroy twice, once through Disposable listening for unload, and once from widget.destroy. I'll fix it in https://bugzilla.mozilla.org/show_bug.cgi?id=864986.

I do like the idea of checking memory after each test, but for now I just want to get this landed and out of my queue.

@Mossop Mossop closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Oct 4, 2012
  1. @Mossop

    Add leak instrumentation

    Mossop authored
  2. @Mossop
  3. @Mossop
  4. @Mossop
  5. @Mossop
  6. @Mossop

    Uninstall installed test add-on

    Mossop authored
  7. @Mossop
  8. @Mossop
  9. @Mossop
Commits on Oct 11, 2012
  1. @Mossop

    Merge branch 'master' into check-memory

    Mossop authored
    Conflicts:
    	lib/sdk/test/harness.js
    	lib/sdk/test/loader.js
Commits on Dec 8, 2012
  1. @Mossop

    Merge branch 'master' into check-memory

    Mossop authored
    Conflicts:
    	test/test-addon-installer.js
    	test/test-content-symbiont.js
Commits on Apr 22, 2013
  1. @Mossop

    Merge branch 'master' into check-memory

    Mossop authored
    Conflicts:
    	lib/sdk/deprecated/unit-test-finder.js
    	lib/sdk/test/harness.js
    	lib/sdk/test/loader.js
    	test/test-addon-installer.js
    	test/test-page-mod.js
  2. @Mossop

    Make memory checking optional.

    Mossop authored
This page is out of date. Refresh to see the latest.
View
1  app-extension/bootstrap.js
@@ -235,6 +235,7 @@ function startup(data, reasonCode) {
stopOnError: options.stopOnError,
verbose: options.verbose,
parseable: options.parseable,
+ checkMemory: options.check_memory,
}
}
});
View
153 lib/sdk/test/harness.js
@@ -57,6 +57,9 @@ var results = {
testRuns: []
};
+// A list of the compartments and windows loaded after startup
+var startLeaks;
+
// JSON serialization of last memory usage stats; we keep it stringified
// so we don't actually change the memory usage stats (in terms of objects)
// of the JSRuntime we're profiling.
@@ -162,9 +165,32 @@ function reportMemoryUsage() {
var gWeakrefInfo;
-function showResults() {
+function checkMemory() {
memory.gc();
+ setTimeout(function () {
+ memory.gc();
+ setTimeout(function () {
+ let leaks = getPotentialLeaks();
+ let compartmentURLs = Object.keys(leaks.compartments).filter(function(url) {
+ return !(url in startLeaks.compartments);
+ });
+
+ let windowURLs = Object.keys(leaks.windows).filter(function(url) {
+ return !(url in startLeaks.windows);
+ });
+
+ for (let url of compartmentURLs)
+ console.warn("LEAKED", leaks.compartments[url]);
+
+ for (let url of windowURLs)
+ console.warn("LEAKED", leaks.windows[url]);
+
+ showResults();
+ });
+ });
+}
+function showResults() {
if (gWeakrefInfo) {
gWeakrefInfo.forEach(
function(info) {
@@ -227,7 +253,7 @@ function cleanup() {
console.exception(e);
};
- setTimeout(showResults, 1);
+ setTimeout(require('@test/options').checkMemory ? checkMemory : showResults, 1);
// dump the coverobject
if (Object.keys(coverObject).length){
@@ -245,6 +271,123 @@ function cleanup() {
}
}
+function getPotentialLeaks() {
+ memory.gc();
+
+ // Things we can assume are part of the platform and so aren't leaks
+ let WHITELIST_BASE_URLS = [
+ "chrome://",
+ "resource:///",
+ "resource://app/",
+ "resource://gre/",
+ "resource://gre-resources/",
+ "resource://pdf.js/",
+ "resource://pdf.js.components/",
+ "resource://services-common/",
+ "resource://services-crypto/",
+ "resource://services-sync/"
+ ];
+
+ let ioService = Cc["@mozilla.org/network/io-service;1"].
+ getService(Ci.nsIIOService);
+ let uri = ioService.newURI("chrome://global/content/", "UTF-8", null);
+ let chromeReg = Cc["@mozilla.org/chrome/chrome-registry;1"].
+ getService(Ci.nsIChromeRegistry);
+ uri = chromeReg.convertChromeURL(uri);
+ let spec = uri.spec;
+ let pos = spec.indexOf("!/");
+ WHITELIST_BASE_URLS.push(spec.substring(0, pos + 2));
+
+ let compartmentRegexp = new RegExp("^explicit/js-non-window/compartments/non-window-global/compartment\\((.+)\\)/");
+ let compartmentDetails = new RegExp("^([^,]+)(?:, (.+?))?(?: \\(from: (.*)\\))?$");
+ let windowRegexp = new RegExp("^explicit/window-objects/top\\((.*)\\)/active");
+ let windowDetails = new RegExp("^(.*), id=.*$");
+
+ function isPossibleLeak(item) {
+ if (!item.location)
+ return false;
+
+ for (let whitelist of WHITELIST_BASE_URLS) {
+ if (item.location.substring(0, whitelist.length) == whitelist)
+ return false;
+ }
+
+ return true;
+ }
+
+ let compartments = {};
+ let windows = {};
+ function logReporter(process, path, kind, units, amount, description) {
+ let matches = compartmentRegexp.exec(path);
+ if (matches) {
+ if (matches[1] in compartments)
+ return;
+
+ let details = compartmentDetails.exec(matches[1]);
+ if (!details) {
+ console.error("Unable to parse compartment detail " + matches[1]);
+ return;
+ }
+
+ let item = {
+ path: matches[1],
+ principal: details[1],
+ location: details[2] ? details[2].replace("\\", "/", "g") : undefined,
+ source: details[3] ? details[3].split(" -> ").reverse() : undefined,
+ toString: function() this.location
+ };
+
+ if (!isPossibleLeak(item))
+ return;
+
+ compartments[matches[1]] = item;
+ return;
+ }
+
+ matches = windowRegexp.exec(path);
+ if (matches) {
+ if (matches[1] in windows)
+ return;
+
+ let details = windowDetails.exec(matches[1]);
+ if (!details) {
+ console.error("Unable to parse window detail " + matches[1]);
+ return;
+ }
+
+ let item = {
+ path: matches[1],
+ location: details[1].replace("\\", "/", "g"),
+ source: [details[1].replace("\\", "/", "g")],
+ toString: function() this.location
+ };
+
+ if (!isPossibleLeak(item))
+ return;
+
+ windows[matches[1]] = item;
+ }
+ }
+
+ let mgr = Cc["@mozilla.org/memory-reporter-manager;1"].
+ getService(Ci.nsIMemoryReporterManager);
+
+ let enm = mgr.enumerateReporters();
+ while (enm.hasMoreElements()) {
+ let reporter = enm.getNext().QueryInterface(Ci.nsIMemoryReporter);
+ logReporter(reporter.process, reporter.path, reporter.kind, reporter.units,
+ reporter.amount, reporter.description);
+ }
+
+ let enm = mgr.enumerateMultiReporters();
+ while (enm.hasMoreElements()) {
+ let mr = enm.getNext().QueryInterface(Ci.nsIMemoryMultiReporter);
+ mr.collectReports(logReporter, null);
+ }
+
+ return { compartments: compartments, windows: windows };
+}
+
function nextIteration(tests) {
if (tests) {
results.passed += tests.passed;
@@ -440,6 +583,12 @@ var runTests = exports.runTests = function runTests(options) {
global: {} // useful for storing things like coverage testing.
});
+ // Load these before getting initial leak stats as they will still be in
+ // memory when we check later
+ require("../deprecated/unit-test");
+ require("../deprecated/unit-test-finder");
+ startLeaks = getPotentialLeaks();
+
nextIteration();
} catch (e) {
let frames = fromException(e).reverse().reduce(function(frames, frame) {
View
5 lib/sdk/test/loader.js
@@ -6,6 +6,7 @@
const { Loader, resolveURI, Require,
unload, override, descriptor } = require('../loader/cuddlefish');
+const { ensure } = require('../system/unload');
const addonWindow = require('../addon/window');
const { PlainTextConsole } = require("sdk/console/plain-text");
@@ -19,7 +20,7 @@ function CustomLoader(module, globals, packaging) {
});
let loader = Loader(options);
- return Object.create(loader, descriptor({
+ let wrapper = Object.create(loader, descriptor({
require: Require(loader, module),
sandbox: function(id) {
let requirement = loader.resolve(id, module.id);
@@ -30,6 +31,8 @@ function CustomLoader(module, globals, packaging) {
unload(loader, reason);
}
}));
+ ensure(wrapper);
+ return wrapper;
};
exports.Loader = CustomLoader;
View
8 python-lib/cuddlefish/__init__.py
@@ -228,6 +228,12 @@
metavar=None,
default=False,
cmds=['sdocs'])),
+ (("", "--check-memory",), dict(dest="check_memory",
+ help="attempts to detect leaked compartments after a test run",
+ action="store_true",
+ default=False,
+ cmds=['test', 'testpkgs', 'testaddons',
+ 'testall'])),
]
),
@@ -660,7 +666,7 @@ def run(arguments=sys.argv[1:], target_cfg=None, pkg_cfg=None,
# a Mozilla application (which includes running tests).
use_main = False
- inherited_options = ['verbose', 'enable_e10s', 'parseable']
+ inherited_options = ['verbose', 'enable_e10s', 'parseable', 'check_memory']
enforce_timeouts = False
if command == "xpi":
Something went wrong with that request. Please try again.