Discuss : possibility to execute standalone (callable in devtools) process ?

[fcamblor]

Is your feature request related to a problem? Please describe.
I was wondering if it was something doable to provide an importable module for single-file, that could make it executable programmatically from DevTools (without having to install the extension)

Something like :

import SingleFile from 'https://cdn.skypack.dev/single-file-standalone'
const capturedHtml = await SingleFile()
// do something with captured html

(this is maybe something already existing / easily achievable, but didn't found it in the doc as the CLI is not matching my need here unfortunately, see in additional context section below)

Describe the solution you'd like
Publish a JS library providing API used to capture current page programmatically.

Additional context (optional)
The extension is fantastic, but in some cases, I would like to trigger it directly from devtools.

Typically, I need to capture HTML generated on an Android device webview (where I can't install the chrome extension, and where I can't get any URL that could be processed by the CLI).
I can remotely inspect the Android webview from my laptop, and can have access to the whole devtools context (including DOM, etc.) from it, but didn't found any way to remotely capture DOM content as single file.

With such standalone library, I assume this would become something feasible.
WDYT ?

(not sure if this would be that easy anyway, as any extension API would have to be avoided in that case)

[gildas-lormeau]

You can achieve this by:

1. Injecting https://raw.githubusercontent.com/gildas-lormeau/SingleFile/master/dist/single-file.js in the webview, it will create a global variable named singlefile
2. Running for example the following code (all the options are set to false by default)

const { content, title, filename } = await singlefile.getPageData({
  removeImports: true,
  removeScripts: true,
  removeAudioSrc: true,
  removeVideoSrc: true  
});

The variable content will contain the HTML content of the page. Frame contents won't be embedded.

Note however that you will need to disable CORS in order to be able to retrieve contents from external domains. Otherwise, it's also possible to pass a custom function used to fetch all the contents. This is how SingleFile solves this issue in the JSDOM implementation, see below

SingleFile/cli/back-ends/jsdom.js

Line 81 in 6b9b6df

const pageData = await win.singlefile.getPageData(options, { fetch: url => fetchResource(url, options) }, doc, win);

SingleFile/cli/back-ends/jsdom.js

Lines 87 to 107 in 6b9b6df

	async function fetchResource(resourceURL) {
	return new Promise((resolve, reject) => {
	const xhrRequest = new win.XMLHttpRequest();
	xhrRequest.withCredentials = true;
	xhrRequest.responseType = "arraybuffer";
	xhrRequest.onerror = event => reject(new Error(event.detail));
	xhrRequest.onreadystatechange = () => {
	if (xhrRequest.readyState == win.XMLHttpRequest.DONE) {
	resolve({
	arrayBuffer: async () => new Uint8Array(xhrRequest.response).buffer,
	headers: {
	get: headerName => xhrRequest.getResponseHeader(headerName)
	},
	status: xhrRequest.status
	});
	}
	};
	xhrRequest.open("GET", resourceURL, true);
	xhrRequest.send();
	});
	}

I hope this information is useful. Feel free to ask me more questions.

[fcamblor]

Hi @gildas-lormeau (and yes, happy to see you again ;-)),

This is perfect, this whole material is very helpful (and is working like a charm, just tested it).
Thanks a lot for that, you don't know how much time you just saved me :)

[gildas-lormeau]

Great! (I edited the message because I wasn't 100% sure you were the right fcamblor ;-)). I am really glad to hear that SingleFile is useful to you.

[fcamblor]

@gildas-lormeau Hello Gildas (and sorry to bother you again ! :-) )

I'm facing an issue on fetchResource impl (from your jsdom example) : when one of my links targets a svg+xml file, this is not working.
By debugging it, I discovered that xhrRequest.readyState was never DONE when using a xhr.responseType = "arraybuffer" on a SVG URL.

I tried to tweak a little bit the implementation (using xhr.responseType="text" and TextEncoder.encode() on the response), but it doesn't seem to work (I continue having a <img src="data:null;base64," /> resolved image in the DOM)

I noticed there was a util.parseSVGContent() utility function but it seems to only be called in xlinks.

FYI, my altered fetchResource impl :

async function fetchResource(resourceURL, win) { 
 	return new Promise(function (resolve, reject) { 
 		const xhrRequest = new win.XMLHttpRequest(); 
 		xhrRequest.withCredentials = true;
 		// Quick & dirty fix for SVG file management (let's make it work first, then clean it)
 		if(resourceURL.substr(resourceURL.length-".svg".length) === '.svg') {
 		    xhrRequest.responseType = "text";
 		} else {
 		    xhrRequest.responseType = "arraybuffer"; 
 		}
 		xhrRequest.onerror = function(event) { 
			console.error(`Got an error while fetching ${resourceURL}: ${JSON.stringify(event)}`);
			return reject(new Error(event.detail)); 
		};
 		xhrRequest.onreadystatechange = function() {
 			if (xhrRequest.readyState == win.XMLHttpRequest.DONE) { 
 				if(xhrRequest.responseType === "text") {
 					console.log("text response type !");
				}
				resolve({ 
					arrayBuffer: function(){
						var ui8Array;
						if(xhrRequest.responseType === 'text') {
							ui8Array = new TextEncoder().encode(xhrRequest.response);
						} else {
							ui8Array = new Uint8Array(xhrRequest.response);
						}
						return ui8Array.buffer;
					}, 
					headers: { 
						get: function(headerName) { return xhrRequest.getResponseHeader(headerName); }
					}, 
					status: xhrRequest.status 
 				}); 
			} 
 		}; 

 		console.debug(`Opening ${resourceURL}...`);
 		xhrRequest.open("GET", resourceURL, true); 
 		xhrRequest.send(); 
 	}); 
 } 

Would you have any piece of advice to share on this ?

[gildas-lormeau]

That's a weird issue... It means the promise returned by fetchResource is never settled? Maybe you could try to log the xhrRequest.readyState values by adding console.log("readyState", xhrRequest.readyState) just before

if (xhrRequest.readyState == win.XMLHttpRequest.DONE) { 

If you can, I would also recommend to attach a debugger to the webview and check the network requests.

[fcamblor]

Hello Gildas,

I was doubly wrong ... SVG files were properly loaded (even with arraybuffer responseType).

I just found the root cause of my issue : I was executing single-file inside a cordova webview where FileReader class was overridden ... but without addEventLIstener() method on it

This was resulting into a silent error here, leading to a data:null;base64 replacement.

Sorry for the false positive and the inconvenience / useless noise.

[gildas-lormeau]

Hello Frédéric,

No problem, I'm really glad to hear this was not in fact a weird issue and that you were able to identify and fix its root cause. A quick and dirty way to circumvent it could be to implement the missing addEventLIstener method, e.g. inject the code below before running SingleFile (not tested):

FileReader.prototype.addEventListener = function(name, callback) { this["on" + name] = callback; };

BTW, if it is possible (including in private if needed), I would be interested to know more about your use case, it sounds interesting :)

[fcamblor]

That's exactly what I did, with a little difference :-)

  if(!FileReader.prototype.addEventListener && eventName) {
    FileReader.prototype.addEventListener = function(eventName, callback, useCapture) {
          // Loosing useCapture flag here... not a big deal at the moment
	  this['on'+eventName.toLowerCase()] = callback;
	};
  }

My use case is : I'm working on a web app aimed at translating web applications.

One of the main features of this webapp is to be able to provide context to translators, and particularly "living apps" where translatable messages reside (that way, they are able to see in realtime if the translation they provided is going to break a screen or not)

i18n.small.demo.mov

What I call a "living app" is typically a self-sufficient single HTML page that he can contribute to the translation app (the blue app in the video above) and this is where single-file comes in : I wrote a small js snippet aimed at capturing self-sufficient HTML page during app browsing, so that my users are able to easily contribute screens to the translation app while navigating on the apps they want to translate (as if he was taking "screenshots" of the app, except that this is a "living screenshot" because we have the whole DOM captured :-))

Given that we're working a lot with Cordova for our mobile apps, I need to make this work on cordova webviews as well, and this is now working since yesterday evening (or, let's say ... today early morning ;-))

[gildas-lormeau]

That's a great use-case! It also sounds like a great user experience. This is not the first time that SingleFile is used in the translation field. Funny, I didn't anticipate it. Anyway, I hope SingleFile will bring you satisfaction. Good luck and don't hesitate to ping me if you need help.