Implement Chrome Extension

[abrichr]

Certain use cases (e.g. SalesForce, LinkedIn) would greatly benefit from reading and writing website markup and the DOM (i.e. by simplifying tokenization of screenshots).

1. Implement a Chrome Plugin that sends and receives messages from PuterBot via native messaging: https://developer.chrome.com/docs/extensions/mv3/nativeMessaging/ . On every change to the Document Object Model (DOM) of the currently opened page, the extension should send a message to PuterBot.

2. Implement an adapter in PuterBot to receive and store the messages with https://github.com/Rayquaza01/nativemessaging. To start with, this can look something like this:

browser.py:

import nativemessaging

def main():
    # based on https://github.com/Rayquaza01/nativemessaging#sample
    reply_num = 0
    while True:
        message = nativemessaging.get_message()
        print(message)
        nativemessaging.send_message(nativemessaging.encode_message(str(reply_num))
        reply_num += 1
if __name__ == "__main__":
    main()

In the extension:

// from https://github.com/Rayquaza01/nativemessaging#sample

function onReceived(response) {
    console.log(response);
}

// runtime.connectNative
var port = browser.runtime.connectNative("application_name");
port.onMessage.addListener(onReceived);
port.postMessage("hello");


// runtime.sendNativeMessage
browser.runtime.sendNativeMessage("puterbot", document).then(onReceived);

// based on https://stackoverflow.com/questions/8882502/how-to-track-dom-change-in-chrome-extension

let observer = new MutationObserver(mutations => {
    for(let mutation of mutations) {
        browser.runtime.sendNativeMessage("puterbot", mutation).then(onReceived);
    }
});
observer.observe(document, { childList: true, subtree: true });

Once this is working, we will want to add the necessary functionality to record.py.

[KrishPatel13]

Regarding MutationObserver: https://stackoverflow.com/questions/20383356/mutation-observer-is-undefined
It is only supported in Internet Explorer 11.

Currently Figuring out an alternative for Chrome.

[abrichr]

It is only supported in Internet Explorer 11.

Are you sure? https://developer.chrome.com/blog/detect-dom-changes-with-mutation-observers/

[KrishPatel13]

Some Notes regarding Chrome Extension: https://docs.google.com/presentation/d/106AXW3sBe7-7E-zIggnMnaUKUXWAj_aAuSxBspTDcGk/edit?usp=sharing

[KrishPatel13]

Link to information on publishing extension: https://developer.chrome.com/docs/webstore/publish/

[abrichr]

#364

[KrishPatel13]

This issue shall be closed as completed if this PR gets successfully merged! @abrichr