Feature/console debugging (#568)

* add linting for no-param-reassign and fix resulting linting errors

* add linting for prefer-object-spread and fix resulting linting errors

* add linting for no-restricted-syntax and fix 1/2 of resulting errors

* add linting for no-prototype-builtins and fix resulting linting errors

* add linting for class-methods-use-this and fix most resulting errors. Add /tools and /tests to linter. Update height in UpgradeBanner

* finish linting for class-methods-use-this

* add linting for no-mixed-operators and fix resulting linting errors

* add linting for import/prefer-default-export and fix resulting linting errors

* add linting for react/no-access-state-in-setstate and fix resulting linting errors

* add linting for react/jsx-props-no-spreading and fix resulting linting errors

* finish linting errors for no-restricted-syntax. 1 remains: couldn't resolve removing iterator loops for urlSearchParams

* Fix linting errors resulting from the merge with develop

* Refactor UNSAFE_componentWillMount into either constructor or componentDidMount, leave notes for how decision was made.

* Refactor UNSAFE_componentWillReceiveProps to componentDidUpdate or getDerivedStateFromProps

* re-enable lint exception for no-prototype-builtins and revert calls back to hasOwnProperty

* add single line exception for no-restricted-syntax linting rule

* add linting for react/destructuring-assignment and fix errors. ToDo: test code and check for errors

* Fix minor bugs

* Fix General Settings last updated text

* rework linting rule no-param-reassign to have more exceptions and param object destructuring

* Remove file and line linting exceptions.

* re-add linting rule react/sort-comp and fix resulting errors

* remove added linting exception consistent-return and fix resulting errors

* remove added linting expression no-use-before-define and fix resulting errors. fix BugDB.js bug.

* Fix linting error

* fix minor bugs

* Code cleanup: fix PromoModal imports

* remove unnecessary hasOwnProperty calls after refactored for...in loops

* Fix missing strings bug

* Fix last remaining string bug

* Begin work on a global object for debugging.

* Add accountEvents array to Ghostery Debug for logging login/out events

* Add way to output window.GHOSTERY for the user

* prevent error when not signed in

* Promises don't reject, just give important data to user

* Download DebugInfo as a JSON file.

* Fix linting error

* Fix bug with race condition on Debug Information page

* Add translation string for opening debug information page

* address some issues on PR

* Address remaining PR issues

* Update Help component test snapshot

* Remove front-end parts of the debugging interface. Debugger will be console only

* Remove log key reference from README

* Remove LOG global and refactor log utility to gate on debug value instead of log value

* Delete unneeded debug_information message handler from background

* Implement logging toggle in console debugger

* Implement globals getter with support for string and regex literal arguments for console debugger

* Add support for hitting AB server with arbitrary ir from debug console

* Reorganize the tab info objects

* Rename getActiveTabInfo to accurately reflect function. Add support for logging that overrides global logging setting

* Add help and status method stubs to GhosteryDebug. Add support for getting all conf props, a single one, or a subset using regexp

* Improve pretty printing for GhosteryDebug help and status methods

* Only expose global GHOSTERY debug option if consoleDebug is set in manifest

* Factor out getting an object slice to getObjectSlice utility function

* Remove the consoleDebug manifest key and related code

* Add support for forcing promo modal display to console debugger

* Rename GHOSTERY console debug object to ghostery because toggling Caps Lock is ANNoyINg

* Fleshing out help function for console debugger

* Prettify ghostery console debugger help function output. Continue fleshing out help contents

* Continue fleshing out debugger help. Add first handling of a case where a function name is passed in to the help method

* Exploring console API to format debugger output better

* Continue iterating on debugger output formatting

* Add ads to console debugger

* Iterate on debugger help output

* Flesh out implementation of ghosteryDebug#getConfData output. Flesh out debugger help output for this function

* Flesh out implementation of ghosteryDebug#getGlobals output and its help output

* Factor pickRandomArrEl function out to utils, flesh it out, and document it. Revise help output for getABTests to conform to standard format for function help output. Remove addition of linebreaks from typeset method to help fix and improve output formatting. Adjust output styles accordingly and define sub header styles.

* Give debugger's typeset method the responsibility for adding new lines before and after debugger help output. Add CSS style markers for a standard way to specify which styles the printToConsole method should apply

* Fix bug in debugger output formatting

* Add debugger help output for ghostery.hitABServerWithIr()

* Begin to factor out help strings from GhosteryDebug#help to a static field on GhosteryDebug

* Factor help strings out to static class fields on GhosteryDebug

* Organize GhosteryDebug by grouping formatting and output code and grouping help CLI & strings. Document GhosteryDebug#printToConsole and GhosteryDebug#typeset. Add ToC.

* Rearrange static class fields in help strings section of GhosteryDebug to account for the apparent fact that static class fields must be defined before being referenced by other class fields. Fix small typos.

* Move toggleLogging to settings GhosteryDebug instance field. Move status to settings.show. Move isLog and objectDisplayStyle into settings

* Add help output for GhosteryDebug#settings.toggleLogging

* Add help output for GhosteryDebug#settings.show(). Standardize show() and toggleLogging() output to use typeset() and printToConsole(). Add Babel support for optional chaining syntax

* Flesh out help output for GhosteryDebug#settings.show

* Begin GhosteryDebug#settings.setOutputStyle implementation. Factor common printToConsole code out to a helper

* Finish implementing GhosteryDebug#settings.toggleOutputStyle. Factor out common code in the settings togglers to helper methods

* Flesh out GhosteryDebugger#forceOnePromoModalDisplay to produce formatted output and handle bad arguments. Rework PromoModals so that modals can be added, removed, and priority-shifted around without necessitating any changes to the GhosteryDebug help output or other code.

* Factor string array assembly out from GhosteryDebugger#help to a helper that can be called independently by other methods. Update showPromoModal output to use the standard formatting provided by the typeset & printToConsole chain. Improve docs for PromoModals#showPromoModal. Make the PromoModals#showPromoModal argument handling case-insensitive. Add PromoModals.getActiveModalTypes. Improve getObjectSlice util function docs. Improve pickRandomArrEl util function docs and error messaging. Add capitalize utility function. Solve global warming.

* Remove some unused or redundant code from GhosteryDebug

* Prep printToConsole and typeset for handling objects, so that printToConsole and its helpers can take sole responsibility for console calls

* Add error handling to getObjectSlice util function and return whether a match was found, so we can provide better messaging about the return values in GhosteryDebug methods that use the utility

* Adjust error check order in getObjectSlice util function

* Remove extra line

* Pipe getConfData and getGlobals output through the standard typeset / printToConsole  chain

* Move GhosteryDebug#_outputObjectSlice out of the printing section since it now delegates printing to printToConsole

* Convert GhosteryDebug#getABTests to use standard typeset and printToConsole output pipeline. Factor pushing an object to output array as object or string based on debugger setting out to helper

* Update ABTest to support hitting the A/B server without automatically logging out the return values

* Implement standardized output formatting and error checking for GhosteryDebug#hitABServerWithIr

* rename hitABServerWithIr to hitABServerWithIrAsync to make it clear that it is beautiful

* Rename hitABServerWithIrAsync to fetchABTestsWithIr cos it is shorter and will match fetchCMPCampaigns

* Replace some magic strings. Clean up fetchABTestsWithIr output formatting

* Rework debugger log toggling to avoid creating a circular dependency

* Replace '__SUBHEADER__' magic string with string constant

* Replace '__MAINHEADER__' magic string with string constant

* Replace '__HIGHLIGHT__' CSS marker magic string with string constant in GhosteryDebugger

* Allow changing values of CSS marker string constants without having to modify GhosteryDebug#printToConsole or printFormatted

* Variety of minor org and doc tweaks to GhosteryDebug

* Minor reorg in GhosteryDebug

* Rework CMP to accept fetch requests from debugger

* Continue implementing GhosteryDebug#fetchCMPCampaigns. Improve documentation for several utility functions.

* Fleshing out documentation for GhosteryDebug fields

* Further document fields in GhosteryDebug

* Flesh out fetchCMPCampaigns implementation to conform to standard output formatting. Add docs

* add openPanel function to debugger for automation testing

* rename debugger class

* rename ghostery debugger vars

* open intro hub and android panel from debugger

* Cut Debugger#getDebugInfo

* Remove babel optional chaining plugin and the couple Debugger lines that used it

* Update naming and itdy up export

* rework getActiveTabInfo and add to help menu

* Refactor Debugger#getUserData to print to console better and to handle errors better

* Convert Debugger#getUserData to an arrow func. JSDoc it.

* Add help output for Debugger#getUserData. Clean up main help menu a little.

* handle undefined returns in console

* Print account events at the end of Debugger#getUserData output

* Cap Debugger accountEvents array at 1000 most recent events

* Update help output for Debugger#getUserData

* Fix typo

* Conceal Debugger _toggleSettingsHelper from console

* Rename Debugger accountEvents array to signal that it's private

* Add JSDocs for Debugger getABTests and getConfData. Tidy up existing JSDocs

* Add Debugger getGlobals JSDoc. Tidy up openPanel and openIntroHub JSDocs

* Add JSDoc for Debugger showPromoModal. Remove @async tag in other ones as it is only intended for use with functions defined with the async keyword

* Add Debugger addAccountEvent JSDoc

* Add JSDoc for Debugger _printObjectSlice

* Add JSDoc for Debugger _push

* JSDoc Debugger settings object and settings.show. Convert settings to a closure to force JSDoc to document inner properties

* JSDoc the Debugger's toggle setting methods

* Fix accountEvents printing in Debugger getUserData

* Fix bug that was preventing subscription data from showing in Debugger getUserData output

* Make it so timestamps are not double quoted in ghostery._accountEvents output

Co-authored-by: Christopher Tino <4699516+christophertino@users.noreply.github.com>
Co-authored-by: wlycdgr <ilya.zarembsky@gmail.com>
Co-authored-by: Christopher Tino <christopher_tino@yahoo.com>

README.md

@@ -51,7 +51,6 @@ $ yarn build.watch
 ```javascript
 // In manifest.json set
 "debug": true,
-"log": true,
 ```
 
 ## Testing and Linting

manifest.json

@@ -1,4 +1,5 @@
 {
+  	"debug": true,
 	"manifest_version": 2,
 	"author": "Ghostery",
 	"name": "__MSG_name__",
@@ -7,7 +8,6 @@
 	"version_name": "8.5.3",
 	"default_locale": "en",
 	"description": "__MSG_short_description__",
-	"log": true,
 	"icons": {
 		"16": "app/images/icon16.png",
 		"48": "app/images/icon48.png",

src/background.js

@@ -17,6 +17,7 @@
 import { debounce, every, size } from 'underscore';
 import moment from 'moment/min/moment-with-locales.min';
 import cliqz, { HUMANWEB_MODULE, HPN_MODULE } from './classes/Cliqz';
+import ghosteryDebugger from './classes/Debugger';
 // object classes
 import Events from './classes/EventHandlers';
 import Policy from './classes/Policy';
@@ -53,6 +54,9 @@ import { sendCliqzModuleCounts } from './utils/cliqzModulesData';
 // module from Developer Tools Console.
 window.CLIQZ = cliqz;
 
+// For debug purposes, provide access to Ghostery's internal data.
+window.ghostery = ghosteryDebugger;
+
 // class instantiation
 const events = new Events();
 // function shortcuts
@@ -1233,7 +1237,7 @@ function initializeDispatcher() {
 }
 
 /**
- * WebRequest pipeline initialisation: find which Cliqz modules are enabled,
+ * WebRequest pipeline initialization: find which Cliqz modules are enabled,
  * add their handlers, then put Ghostery event handlers before them all.
  * If Cliqz modules are subsequently enabled, their event handlers will always
  * be added after Ghostery's.
@@ -1784,6 +1788,7 @@ function init() {
 			account.migrate()
 				.then(() => {
 					if (conf.account !== null) {
+						ghosteryDebugger.addAccountEvent('app started', 'signed in', conf.account);
 						return account.getUser()
 							.then(account.getUserSettings)
 							.then(() => {
@@ -1793,6 +1798,7 @@ function init() {
 								return false;
 							});
 					}
+					ghosteryDebugger.addAccountEvent('app started', 'not signed in');
 					if (globals.JUST_INSTALLED) {
 						setGhosteryDefaultBlocking();
 					}

src/classes/ABTest.js

@@ -38,44 +38,71 @@ class ABTest {
 		return this.tests.hasOwnProperty(name);
 	}
 
+	/**
+	 * Return the tests object
+	 * @return {Object}
+	 */
+	getTests() {
+		return this.tests;
+	}
+
 	/**
 	 * Send parameters to A/B Test server and receive tests data.
-	 * @return {Promise} 		dictionary with all tests to be executed
+	 * @param	{Number} irDebugOverride		optional. supports hitting AB server with custom ir from debug console
+	 * @return {Promise} 						dictionary with all tests to be executed
 	 */
-	fetch() {
+	fetch(irDebugOverride) {
 		log('A/B Tests: fetching...');
 
-		const URL = `${CMP_BASE_URL}/abtestcheck
-			?os=${encodeURIComponent(BROWSER_INFO.os)}
-			&install_date=${encodeURIComponent(conf.install_date)}
-			&ir=${encodeURIComponent(conf.install_random_number)}
-			&gv=${encodeURIComponent(EXTENSION_VERSION)}
-			&si=${conf.account ? '1' : '0'}
-			&ua=${encodeURIComponent(BROWSER_INFO.name)}
-			&v=${encodeURIComponent(conf.cmp_version)}
-			&l=${encodeURIComponent(conf.language)}`;
+		const URL = ABTest._buildURL(irDebugOverride);
 
 		return getJson(URL).then((data) => {
 			if (data && Array.isArray(data)) {
 				log('A/B Tests: fetched', JSON.stringify(data));
-				// merge all tests into this.tests object
-				// this will overwrite all previous tests
-				this.tests = data.reduce(
-					(tests, test) => Object.assign(tests, { [test.name]: test.data }),
-					{}
-				);
-				this.hasBeenFetched = true;
+				this._updateTests(data);
+				log('A/B Tests: tests updated to', this.getTests());
 			} else {
 				log('A/B Tests: no tests found.');
 			}
-
-			// update conf
-			globals.SESSION.abtests = this.tests;
-			log('A/B Tests: tests updated to', JSON.stringify(this.tests));
 		}).catch(() => {
 			log('A/B Tests: error fetching.');
 		});
 	}
+
+	silentFetch(ir) {
+		const URL = ABTest._buildURL(ir);
+
+		return getJson(URL).then((data) => {
+			if (data && Array.isArray(data)) {
+				this._updateTests(data);
+			}
+			return 'resolved';
+		}).catch(() => 'rejected');
+	}
+
+	static _buildURL(ir) {
+		return (`${CMP_BASE_URL}/abtestcheck
+			?os=${encodeURIComponent(BROWSER_INFO.os)}
+			&install_date=${encodeURIComponent(conf.install_date)}
+			&ir=${encodeURIComponent((typeof ir === 'number') ? ir : conf.install_random_number)}
+			&gv=${encodeURIComponent(EXTENSION_VERSION)}
+			&si=${conf.account ? '1' : '0'}
+			&ua=${encodeURIComponent(BROWSER_INFO.name)}
+			&v=${encodeURIComponent(conf.cmp_version)}
+			&l=${encodeURIComponent(conf.language)}`
+		);
+	}
+
+	_updateTests(data) {
+		// merge all tests into this.tests object
+		// this will overwrite all previous tests
+		this.tests = data.reduce(
+			(tests, test) => Object.assign(tests, { [test.name]: test.data }),
+			{}
+		);
+		// update conf
+		globals.SESSION.abtests = this.tests;
+	}
 }
 
 // Return the class as a singleton

src/classes/Account.js

@@ -22,6 +22,7 @@ import conf from './Conf';
 import dispatcher from './Dispatcher';
 import { log } from '../utils/common';
 import Api from '../utils/api';
+import ghosteryDebugger from './Debugger';
 
 const api = new Api();
 const {
@@ -83,6 +84,7 @@ class Account {
 			if (res.status >= 400) {
 				return res.json();
 			}
+			ghosteryDebugger.addAccountEvent('login', 'cookie set by fetch POST');
 			this._getUserIDFromCookie().then((userID) => {
 				this._setAccountInfo(userID);
 				this.getUserSubscriptionData();
@@ -103,6 +105,7 @@ class Account {
 			credentials: 'include',
 		}).then((res) => {
 			if (res.status >= 400) {
+				ghosteryDebugger.addAccountEvent('register', 'cookie set by fetch POST');
 				return res.json();
 			}
 			this._getUserIDFromCookie().then((userID) => {
@@ -124,7 +127,10 @@ class Account {
 					credentials: 'include',
 					headers: { 'X-CSRF-Token': cookie.value },
 				}).then((res) => {
-					if (res.status < 400) { return resolve(); }
+					if (res.status < 400) {
+						ghosteryDebugger.addAccountEvent('logout', 'cookie set by fetch POST');
+						return resolve();
+					}
 					return res.json().then(json => reject(json));
 				}).catch(err => reject(err));
 			});
@@ -254,29 +260,34 @@ class Account {
 
 	migrate = () => (
 		new Promise((resolve) => {
+			ghosteryDebugger.addAccountEvent('migrate', 'migrate start');
 			const legacyLoginInfoKey = 'login_info';
 			chrome.storage.local.get(legacyLoginInfoKey, (items) => {
 				if (chrome.runtime.lastError) {
+					ghosteryDebugger.addAccountEvent('migrate', 'runtime error');
 					resolve(new Error(chrome.runtime.lastError));
 					return;
 				}
 
 				const { login_info } = items;
 				if (!items || !login_info) {
+					ghosteryDebugger.addAccountEvent('migrate', 'no items found');
 					resolve();
 					return;
 				}
 
 				// ensure we have all the necessary info
 				const { decoded_user_token, user_token } = login_info;
 				if (!decoded_user_token || !user_token) {
+					ghosteryDebugger.addAccountEvent('migrate', 'found items, not enough info I');
 					chrome.storage.local.remove(legacyLoginInfoKey, () => resolve());
 					return;
 				}
 				const {
 					UserId, csrf_token, RefreshToken, exp
 				} = decoded_user_token;
 				if (!UserId || !csrf_token || !RefreshToken || !exp) {
+					ghosteryDebugger.addAccountEvent('migrate', 'found items, not enough info II');
 					chrome.storage.local.remove(legacyLoginInfoKey, () => resolve());
 					return;
 				}
@@ -311,8 +322,10 @@ class Account {
 					// login
 					this._setAccountInfo(UserId);
 					this.getUserSubscriptionData();
+					ghosteryDebugger.addAccountEvent('migrate', 'remove legacy items');
 					chrome.storage.local.remove(legacyLoginInfoKey, () => resolve());
 				}).catch((err) => {
+					ghosteryDebugger.addAccountEvent('migrate', 'cookies set error');
 					resolve(err);
 				});
 			});

src/classes/CMP.js

@@ -36,36 +36,11 @@ class CMP {
 			return Promise.resolve(false);
 		}
 
-		const URL = `${CMP_BASE_URL}/check
-			?os=${encodeURIComponent(BROWSER_INFO.os)}
-			&offers=${encodeURIComponent(conf.enable_offers ? '1' : '0')}
-			&hw=${encodeURIComponent(conf.enable_human_web ? '1' : '0')}
-			&install_date=${encodeURIComponent(conf.install_date)}
-			&ir=${encodeURIComponent(conf.install_random_number)}
-			&gv=${encodeURIComponent(EXTENSION_VERSION)}
-			&si=${encodeURIComponent(conf.account ? '1' : '0')}
-			&ua=${encodeURIComponent(BROWSER_INFO.name)}
-			&lc=${encodeURIComponent(conf.last_cmp_date)}
-			&v=${encodeURIComponent(conf.cmp_version)}
-			&l=${encodeURIComponent(conf.language)}`;
+		const URL = CMP._buildUrl();
 
 		return getJson(URL).then((data) => {
-			if (data && (!conf.cmp_version || data.Version > conf.cmp_version)) {
-				// set default dismiss
-				data.Campaigns.forEach((dataEntry) => {
-					if (dataEntry.Dismiss === 0) {
-						dataEntry.Dismiss = 10;
-					}
-
-					// set last campaign (dataEntry) run timestamp to avoid running campaigns more than once
-					if (!conf.last_cmp_date || conf.last_cmp_date < dataEntry.Timestamp) {
-						conf.last_cmp_date = dataEntry.Timestamp;
-					}
-				});
-				// update Conf and local CMP_DATA
-				conf.cmp_version = data.Version;
-				globals.SESSION.cmp_data = data.Campaigns;
-				this.CMP_DATA = data.Campaigns;
+			if (CMP._isNewData(data)) {
+				this._updateCampaigns(data);
 				return this.CMP_DATA;
 			}
 			// getJson() returned a 204, meaning no new campaigns available
@@ -77,6 +52,59 @@ class CMP {
 			return false;
 		});
 	}
+
+	debugFetch() {
+		const URL = CMP._buildUrl();
+
+		return getJson(URL)
+			.then((data) => {
+				if (CMP._isNewData(data)) {
+					this._updateCampaigns(data);
+					return ({ ok: true, testsUpdated: true });
+				}
+				globals.SESSION.cmp_data = [];
+				return ({ ok: true, testsUpdated: false });
+			})
+			.catch(() => ({ ok: false, testsUpdated: false }));
+	}
+
+	_updateCampaigns(data) {
+		// set default dismiss
+		data.Campaigns.forEach((dataEntry) => {
+			if (dataEntry.Dismiss === 0) {
+				dataEntry.Dismiss = 10;
+			}
+
+			// set last campaign (dataEntry) run timestamp to avoid running campaigns more than once
+			if (!conf.last_cmp_date || conf.last_cmp_date < dataEntry.Timestamp) {
+				conf.last_cmp_date = dataEntry.Timestamp;
+			}
+		});
+		// update Conf and local CMP_DATA
+		conf.cmp_version = data.Version;
+		globals.SESSION.cmp_data = data.Campaigns;
+		this.CMP_DATA = data.Campaigns;
+	}
+
+	static _buildUrl() {
+		return (`${CMP_BASE_URL}/check
+			?os=${encodeURIComponent(BROWSER_INFO.os)}
+			&offers=${encodeURIComponent(conf.enable_offers ? '1' : '0')}
+			&hw=${encodeURIComponent(conf.enable_human_web ? '1' : '0')}
+			&install_date=${encodeURIComponent(conf.install_date)}
+			&ir=${encodeURIComponent(conf.install_random_number)}
+			&gv=${encodeURIComponent(EXTENSION_VERSION)}
+			&si=${encodeURIComponent(conf.account ? '1' : '0')}
+			&ua=${encodeURIComponent(BROWSER_INFO.name)}
+			&lc=${encodeURIComponent(conf.last_cmp_date)}
+			&v=${encodeURIComponent(conf.cmp_version)}
+			&l=${encodeURIComponent(conf.language)}`
+		);
+	}
+
+	static _isNewData(data) {
+		return (data && (!conf.cmp_version || data.Version > conf.cmp_version));
+	}
 }
 
 // return the class as a singleton