Google Docs will now use canvas based rendering: this may impact some Chrome extensions

[RobertJGabriel]

Just bring it up as and issue and will be willing to help on any develop to get it ready.

Here is the canvas based example https://docs.google.com/document/d/1N1XaAI4ZlCUHNWJBXJUBFjxSTlsD5XctCz6LB3Calcg/preview

@menicosia @ken107 @bboydflo @Amaimersion @JensPLarsen

[ken107]

Thank you, don't look like there is any workaround. Will have to build an actual Google Workspace addon.

[Amaimersion]

Maybe it is only for /preview, not for /edit? I mean, see URL.

For /preview it makes sense, because it is only preview and visitor shouldn't have ability to change its content, even through HTML. It is hard to change content in external canvas.

For me, at the moment, /edit page uses HTML editor, not canvas editor.

But yes, if there will be canvas-rendering instead of HTML-rendering, then it will be a problem.

[Amaimersion]

How you created this preview? Can you provide steps?

[JensPLarsen]

The original post from Google can be found here: https://workspaceupdates.googleblog.com/2021/05/Google-Docs-Canvas-Based-Rendering-Update.html

Do they support any kind of accessibility API with the new design?

[JensPLarsen]

An discussion can be found here: https://news.ycombinator.com/item?id=27129858

[JensPLarsen]

Google have updated their post and opened a small possibility.

If you open "Accessibility Settings" --> "Turn on Screen reader support", Google Docs will emit Readable HTML with the actual text.
Only problem is, this means a complete re-write of the core Google Docs Util code, due to the new HTML structure is different.

If possible the Google Docs Util code should:

• detect if "Screen reader support" is turned on
• have a option to turn "Screen reader support" on from code
• use the new HTML structure

[Amaimersion]

Thank you @JensPLarsen

Do they support any kind of accessibility API with the new design?

I suppose no. For external JS, which didn't create the canvas, it is very hard to interact with 2D context of canvas (I mean, CRUD operations with canvas content). For example, Yandex.Disk Word editor uses canvas based rendering and for me it wasn't possible to somehow interact with document content.

will emit Readable HTML with the actual text.

The problem here is that this provides only ability to read document content. But this library need to have all CRUD operations in order to provide all already implemented functionality. Sure, I will check possibility to interact with document through that "small possibility", but highly unlikely that it will provide all needed things to support this project.

[Amaimersion]

So, I think this project will die when Google Docs will release the canvas based rendering feature. Unfortunately, at the moment it doesn't look like there is anything that can be done about it

[JensPLarsen]

The problem here is that this provides only ability to read document content. But this library need to have all CRUD operations in order to provide all already implemented functionality. Sure, I will check possibility to interact with document through that "small possibility", but highly unlikely that it will provide all needed things to support this project.

So, I think this project will die when Google Docs will release the canvas based rendering feature. Unfortunately, at the moment it doesn't look like there is anything that can be done about it

I agree, if anything it would most likely result in a new project which contains a subset of what this can.

And I fear a new project may have the same issue when Google Docs changes to use WebAssembly (or something else) and everything changes again in X years.

[hudson-dev]

Are there any alternatives to this library that work with with the canvas based rendering, or are there plans to update the library?

[Amaimersion]

Are there any alternatives to this library?

No, as I'm aware.

Are there plans to update the library?

No, at the moment.

[hudson-dev]

Darn that sucks.

[hudson-dev]

How is Grammarly doing it then? https://chrome.google.com/webstore/detail/grammarly-for-chrome/kbfnbcaeplbcioakkpcpgfkobkghlhen?hl=en

[Amaimersion]

Google provided temporary support for such extensions. If the extension needs to interact with a document through DOM, then the extension can force Google Docs to use HTML-based rendering instead of canvas-based rendering.

It is controller via _docs_force_html_by_ext variable:

In that case Google Docs will use HTML instead of canvas.

_docs_force_html_by_ext is undefined:

_docs_force_html_by_ext is set:

However, only whitelisted extensions can use this _docs_force_html_by_ext. Most likely Google Docs team will contact with extension developer to notify him about this feature (as they did this to me).

But anyway, this feature is just a temporary workaround to give developers some time to adapt their extensions. This feature will be disabled soon, maybe in 2021, so it is not reliable.

After that we will see which extensions are able to interact with Google Docs through canvas.

[Amaimersion]

According to my above answer. If you want to use this library, you should install extension which enables HTML-based rendering instead of canvas-based rendering: Grammarly, Smart Copy, etc.

[hudson-dev]

I see, I'll try and contact Google to get whitelisted, although having to install a second extension just to use mine wouldn't be very practical for users.

[Omegastick]

You don't actually need to be whitelisted or install any other extensions. You can force html rendering by adding ?mode=html to the query parameters.

[Amaimersion]

Confirm 👍 Although Google clearly specifies that HTML fallback option has been deprecated and will slowly be removed from production.

[gzomer]

Thanks! @Amaimersion Does Google mention any specific date? Where do they mention it will be removed from production?

[Amaimersion]

They mention it through email. Emails are send to those who is subscribed to https://sites.google.com/corp/google.com/docs-canvas-migration/home

They planning to remove it completely to the end of February.

[Omegastick]

They planning to remove it completely to the end of February.

Sad times.

[gzomer]

Sad indeed : /

[gzomer]

Wait a minute: @Amaimersion Do you know how did Grammarly make it work on canvas?

They are not using the whitelisting anymore, if you inspect the DOM when Grammarly is enabled you can see it works on canvas. I have tried forcing ?mode=html and it also works. Which means Grammarly somehow managed to make it work to read the text from canvas. Now the question is, how?

Grammarly using Canvas

Grammarly using DOM

[gzomer]

I have just downloaded the source code from the Grammarly extension and I found some interesting stuff there. For instance, there is getText function
https://gist.github.com/gzomer/2b809174ce380fced61040005a9a9576#file-grammarly-gdocscanvasinjectedcs-js-L1060

They have a file named Grammarly-gDocsInjectedCs.js which seems to be for the DOM version. But now they have a new file named Grammarly-gDocsCanvasInjectedCs.js (see link above).

I have used this extension to get Grammarly source code https://chrome.google.com/webstore/detail/chrome-extension-source-v/jifpbeccnghkjeaalbbjmodiffmgedin?hl=en

[RobertJGabriel]

@gzomer this is interesting, did you happen to get it to work of the grammarly extension?

[ken107]

That code is a bit complicated but if we can put a breakpoint in that getText function it will be clear

[gzomer]

I was able to partially get the full text. On the onRender function here you can just call n.getText({}) and it will return the full text. You can also get a full document structure by inspecting the variable o.

However, there is one downside. I couldn't make it work without Grammarly extension enabled. There is a sort of a connection between docs and Grammarly const t = document.documentElement.dataset.grGdcConnId || (document.documentElement.dataset.grGdcConnId which seems to happen in another file, but I could understand how does it work.

So it seems to be possible, we just need to figure out how.

I have pushed the whole source code here: https://github.com/gzomer/grammarly-extension

So far the ones that seem to be relevant are:
https://github.com/gzomer/grammarly-extension/blob/main/src/js/Grammarly-gDocsEarlyInjectedCs.js
https://github.com/gzomer/grammarly-extension/blob/main/src/js/Grammarly-gDocsCanvasInjectedCs.js
https://github.com/gzomer/grammarly-extension/blob/main/src/js/Grammarly-gDocs.js

[Omegastick]

I think this function gets the relevant elements for extracting text.

Then this function decodes it into usable text.

Neither of those steps looks trivial.

[ken107]

@Omegastick I use Chrome DevTools to put a breakpoint in that content script function ce(e). That function recursively searches the properties of e to look for the document's text. The question then is where e comes from.

It turns out e is the global variable window.KX_kixApp. If you open Google Docs and press F12, then type into the console window.KX_kixApp you will see that variable.

That variable isn't accessible from the content script's context. I'm not sure how they are able to access it from their content script. The only way I know how to do something like that is by adding a script tag that will execute in the page's JavaScript context, JSON.stringify that variable and pass it to the content script via postMessage. But maybe they're doing some other way.

Edit: ah, got it. The bulk of their scripts executes in the page's JS context. The content script is Grammarly-gDocsEarlyInjector.js, which creates the script tag to inject their scripts into the page's context. I'll see if I can make a proof of concept.

Edit: and the statement on line 943 is how they search for the text. le(n, ((e,t)=>t && "�" === t.toString().charAt(0)), 5) means look for string properties up to 5 levels of depth that begins with that special unicode character.

[redacuve]

@tylertaewook i was able to use annotate canvas with:

manifest.json
        "content_scripts": [
...
      {
        "matches": ["*://docs.google.com/*"],
        "all_frames": false,
        "run_at": "document_start",
        "js": ["js/docs-contentscript.js"]
      }
...
  ],
  "web_accessible_resources": [
    {
      "resources": ["js/docs-contentscript.js", "js/docs-canvas.js"],
      "matches": ["<all_urls>"]
    }
  ],
....
  "permissions": [
    "storage", "scripting"
  ],
...

docs-contentscript.js
var s = document.createElement("script");
s.src = chrome.runtime.getURL("js/docs-canvas.js");
s.onload = function () {
  this.remove();
};
(document.head || document.documentElement).appendChild(s);

docs-canvas.js
window._docs_annotate_canvas_by_ext = "clfoagbcljppiogigjclfjnjpcijnijl";

[dan-online]

The main issue about this, is that these are temporary solutions that once discovered by the google docs team, will be patched or changed again 😓

[kant01ne]

Has anyone been using one of the IDs from that list in prod already? Or even better, do you know how to get whitelisted?

[lukesanborn]

I've been following the thread and its getting hard to keep track of what works. Is there any solution that is not a temporary hack and functions like Grammarly?

[RafaOstrovskiy]

@kant01ne https://stackoverflow.com/a/71321036

[dbjpanda]

Now we can directly add world:MAIN inside your Manifest.json to inject the script into the document head.

manifest.json

"content_scripts": [
   {
      "matches": ["*://docs.google.com/document/*"],
      "run_at": "document_start",
      "js": ["gdocs.js"],
      "world": "MAIN"
    }
  ]

gdocs.js

(() => {
    window._docs_annotate_canvas_by_ext = "npnbdojkgkbcdfdjlfdmplppdphlhhcf"; // Your extension id that is whitelisted by Google 
})();

To whitelist your extension you need to apply here

However, there is a trick, if your extension is not yet whitelisted or pending or rejected you still can use the id of one of the whitelisted extensions. Because Google does not exactly validate it. A list of all Whitelisted extensions can be found from here. Open it's source code and search i8d=

But my question is how do you proceed after you enable annotated canvas. E.g

1. How do you extract the text efficiently,
2. How do you attach a listener on selection of text,
3. How do you find the cursor position?
4. What about selection, range etc?

Does this library works for annotated canvas or is there any other library somewhere?

If anyone have even some code sample to achieve any of the above tasks, kindly help.

[lionelhorn]

For any one interested, I made a list of the whitelisted ids with most downloads.
https://gist.github.com/lionelhorn/1b2d8aa26fa222d2a6803c246640696b

Here is a sample

url	title	count
https://chrome.google.com/webstore/detail/kbfnbcaeplbcioakkpcpgfkobkghlhen	Grammarly: Grammar Checker and Writing App	10000000
https://chrome.google.com/webstore/detail/inoeonmfapjbbkmdafoankkfajkcphgd	Read&Write for Google Chrome™	10000000
https://chrome.google.com/webstore/detail/hjngolefdpdnooamgdldlkjgmdcmcjnc	Equatio - Math made digital	4000000
https://chrome.google.com/webstore/detail/hdhinadidafjejdhmfkjgnolgimiaplp	Read Aloud: A Text to Speech Voice Reader	4000000
https://chrome.google.com/webstore/detail/mloajfnmjckfjbeeofcdaecbelnblden	Snap&Read	3000000
https://chrome.google.com/webstore/detail/ifajfiofeifbbhbionejdliodenmecna	Co:Writer	3000000
https://chrome.google.com/webstore/detail/nllcnknpjnininklegdoijpljgdjkijc	Wordtune - AI-powered Writing Companion	2000000

[RobertJGabriel]

Just for anyone who applys to get white listed. Mine got approved but I was never notified of it.

[grnqrtr]

I'm not exactly sure what this repo is for, but I stumbled upon it trying to figure out how to force google docs into html render mode, and I ended up giving the code from @dbjpanda above to ChatGPT and it helped me make this userscript that can be run with Tampermonkey to solve the issue!

Just in case it helps anyone else that stumbles here:

// ==UserScript==
// @name         Google Docs Script Injector
// @namespace    http://tampermonkey.net/
// @version      1.0
// @description  Inject a script into Google Docs
// @match        *://docs.google.com/document/*
// @run-at       document-start
// @grant        none
// ==/UserScript==

(function() {
    'use strict';

    // Your extension ID that is whitelisted by Google
    const extensionId = "npnbdojkgkbcdfdjlfdmplppdphlhhcf";

    // Inject the script into the page
    const script = document.createElement('script');
    script.innerHTML = `
        (() => {
            window._docs_annotate_canvas_by_ext = "${extensionId}";
        })();
    `;
    document.documentElement.appendChild(script);
})();

[dbjpanda]

I finally able to achieve it and built a chrome extension that works on Google Docs https://chrome.google.com/webstore/detail/chatgpt-chrome-extension/gabfffndnhbbjlgmbogpfaciajcbpkdn

[Georg7]

I finally able to achieve it and built a chrome extension that works on Google Docs https://chrome.google.com/webstore/detail/chatgpt-chrome-extension/gabfffndnhbbjlgmbogpfaciajcbpkdn

Can you share how you got this library to work?

[Georg7]

Anyone willing to share how they're able to get this library to work. @dbjpanda @lionelhorn

I'm able to get docs to render the svgs. Can access a few things like getEditorElement and getCursorElement.

However, I'm not able to get the selected text. I've tried:

Doesn't trigger anything when text is selected

document.addEventListener("selectionchange", function (event) {
  console.log("selectionchange", event);
});

GoogleDocsUtils.getSelection(); returns an empty array

const linesData = GoogleDocsUtils.getSelection();
let selectionData = null;

for (const lineData of linesData) {
  if (lineData) {
    selectionData = lineData;
    break;
  }
}

if (selectionData) {
  console.log("linesData", selectionData.selectedText);
}

[jakobsturm]

I am trying to use a whitelisted ID but can not see any difference in how the canvas is rendered. Could it be that Google implemented a fix that compares the extension ID with the ID in the code?

Also how long does it take to get whitelisted?

[eliChing]

I don't think changing google doc rendering mode will work now, cause after checking the nodes with some grammar check extensions like 'Grammarly' and 'LanguageTool', google doc is still rendering by canvas.

These two nodes include information about the text position and selection position(just position) in Google Docs. you can get the entire text by splice the value of the 'aria-label' attribute
However, I have not yet found a way to get the user-selected text or to highlight specific text based on string index.

[ElijZhang]

@RobertJGabriel Hello, could you please tell me how long does it take to get whitelisted? I applied it two days ago, but my extension id is not in whitelist yet.

[komms]

This function is self-contained:

function le(e, t, n, o=Object.getOwnPropertyNames(e)) {
        const r = new Set
          , i = [];
        let s = 0;
        const a = (o,l,c,u=0)=>{
            if (s++,
            "prototype" === o || l instanceof Window)
                return;
            if (u > n)
                return;
            const d = [...c, o];
            try {
                if (t(o, l))
                    return void i.push({
                        path: d,
                        value: l
                    })
            } catch (e) {}
            var g;
            if (null != l && !r.has(l))
                if (r.add(l),
                Array.isArray(l))
                    l.forEach(((e,t)=>{
                        try {
                            a(t.toString(), e, d, u + 1)
                        } catch (e) {}
                    }
                    ));
                else if (l instanceof Object) {
                    ((g = l) && null !== g && 1 === g.nodeType && "string" == typeof g.nodeName ? Object.getOwnPropertyNames(e).filter((e=>!J.has(e))) : Object.getOwnPropertyNames(l)).forEach((e=>{
                        try {
                            a(e, l[e], d, u + 1)
                        } catch (e) {}
                    }
                    ))
                }
        }
        ;
        return o.forEach((t=>{
            try {
                a(t, e[t], [])
            } catch (e) {}
        }
        )),
        {
            results: i,
            iterations: s
        }
    }

Calling it like this will return the text of the document:

le(window.KX_kixApp, ((e,t)=>t && "\x03" === t.toString().charAt(0)), 5)

Edit: and a de-obfuscated version https://gist.github.com/ken107/2b40c87fcdf27171a5a5fdc489639300

The link is not working. Would love to see the code in the link.

[jakobsturm]

@RobertJGabriel Hello, could you please tell me how long does it take to get whitelisted? I applied it two days ago, but my extension id is not in whitelist yet.

Mine took around three weeks to be whitelisted 🙈

[komms]

@RobertJGabriel Hello, could you please tell me how long does it take to get whitelisted? I applied it two days ago, but my extension id is not in whitelist yet.

Mine took around three weeks to be whitelisted 🙈

How to get whitelisted? What to expect after whitelisting?

[dbjpanda]

This is not that easy. But here are some hints. If you have doubt please reach out to me through linkedin.com/in/dbjpanda

I am just pasting some hints. So if you copy paste it might not work. Try to understand the snippets.

Basically you need to create a exact overlay of the rect element and put it on top of that rect element to get the cursor point by enabling the pointer event.

    const cursor = document.querySelector('#kix-current-user-cursor-caret');
    const cursorBbox = cursor.getBoundingClientRect();
    const x = Math.floor(cursorBbox.right);
    const y = Math.floor(cursorBbox.top);
    const rect = this.getRect(x, y);
        
     getRect(x, y) {
        if (!this.styleElement) {
            this.styleElement = document.createElement('style');
            this.styleElement.id = "enable-pointer-events-on-rect";
            this.styleElement.textContent = [
                `.kix-canvas-tile-content{pointer-events:none!important;}`,
                `#kix-current-user-cursor-caret{pointer-events:none!important;}`,
                `.kix-canvas-tile-content svg>g>rect{pointer-events:all!important; stroke-width:7px !important;}`,
            ].join('\n');

            const parent = document.head || document.documentElement;
            if (parent !== null) {
                parent.appendChild(this.styleElement);
            }
        }

        this.styleElement.disabled = false;
        const rect = document.elementFromPoint(x, y);
        this.styleElement.disabled = true;

        return rect;
    }

    getCaretIndex(rect, x, y) {
        const text = rect.getAttribute('aria-label');
        const textNode = document.createTextNode(text);
        const textElement = this.createTextOverlay(rect, text, textNode);

        if (!text || !textElement || !textNode) return null;

        let range = document.createRange();
        let start = 0;
        let end = textNode.nodeValue.length;

        while (end - start > 1) {
            const mid = Math.floor((start + end) / 2);
            range.setStart(textNode, mid);
            range.setEnd(textNode, end);
            const rects = range.getClientRects();
            if (this.isPointInAnyRect(x, y, rects)) {
                start = mid;
            } else {
                if (x > range.getClientRects()[0].right) {
                    start = end;
                } else {
                    end = mid;
                }
            }
        }

        const caretIndex = start;
        textElement.remove();
        return caretIndex;
    }
    
   createTextOverlay(rect, text, textNode) {
        if (!rect || rect.tagName !== 'rect') return {};

        const textElement = document.createElementNS('http://www.w3.org/2000/svg', 'text');
        const transform = rect.getAttribute('transform') || '';
        const font = rect.getAttribute('data-font-css') || '';

        textElement.setAttribute('x', rect.getAttribute('x'));
        textElement.setAttribute('y', rect.getAttribute('y'));
        textElement.appendChild(textNode);
        textElement.style.setProperty('all', 'initial', 'important');
        textElement.style.setProperty('transform', transform, 'important');
        textElement.style.setProperty('font', font, 'important');
        textElement.style.setProperty('text-anchor', 'start', 'important');

        rect.parentNode.appendChild(textElement);

        const elementRect = rect.getBoundingClientRect();
        const textRect = textElement.getBoundingClientRect();
        const yOffset = ((elementRect.top - textRect.top) + (elementRect.bottom - textRect.bottom)) * 0.5;
        textElement.style.setProperty('transform', `translate(0px,${yOffset}px) ${transform}`, 'important');

        return textElement;
    }

    isPointInAnyRect(x, y, rects) {
        for (const rect of rects) {
            if (x >= Math.floor(rect.left) && x <= Math.floor(rect.right) &&
                y >= Math.floor(rect.top) && y <= Math.floor(rect.bottom)) {
                return true;
            }
        }
        return false;
    }    
            

[jakobsturm]

@RobertJGabriel Hello, could you please tell me how long does it take to get whitelisted? I applied it two days ago, but my extension id is not in whitelist yet.

Mine took around three weeks to be whitelisted 🙈

How to get whitelisted? What to expect after whitelisting?

You can get whitelisted here:
https://docs.google.com/forms/d/e/1FAIpQLScFxMgvXlq2KMsp0UIM66pvThTF1hpojiXQTqyq9txW79OWag/viewform

Once you are whitelisted you will be able to use your extension ID to convert the canvas to a readable DOM.

Important: Google will not inform you once you are whiltelisted, you need to constantly try it to see whether you are whitelisted yet. Previous answers include IDs that you can use to test it while you wait for your extension to be whitelisted

[komms]

This is not that easy. But here are some hints. If you have doubt please reach out to me through linkedin.com/in/dbjpanda

I am just pasting some hints. So if you copy paste it might not work. Try to understand the snippets.

Basically you need to create a exact overlay of the rect element and put it on top of that rect element to get the cursor point by enabling the pointer event.

    const cursor = document.querySelector('#kix-current-user-cursor-caret');
    const cursorBbox = cursor.getBoundingClientRect();
    const x = Math.floor(cursorBbox.right);
    const y = Math.floor(cursorBbox.top);
    const rect = this.getRect(x, y);
        
     getRect(x, y) {
        if (!this.styleElement) {
            this.styleElement = document.createElement('style');
            this.styleElement.id = "enable-pointer-events-on-rect";
            this.styleElement.textContent = [
                `.kix-canvas-tile-content{pointer-events:none!important;}`,
                `#kix-current-user-cursor-caret{pointer-events:none!important;}`,
                `.kix-canvas-tile-content svg>g>rect{pointer-events:all!important; stroke-width:7px !important;}`,
            ].join('\n');

            const parent = document.head || document.documentElement;
            if (parent !== null) {
                parent.appendChild(this.styleElement);
            }
        }

        this.styleElement.disabled = false;
        const rect = document.elementFromPoint(x, y);
        this.styleElement.disabled = true;

        return rect;
    }

    getCaretIndex(rect, x, y) {
        const text = rect.getAttribute('aria-label');
        const textNode = document.createTextNode(text);
        const textElement = this.createTextOverlay(rect, text, textNode);

        if (!text || !textElement || !textNode) return null;

        let range = document.createRange();
        let start = 0;
        let end = textNode.nodeValue.length;

        while (end - start > 1) {
            const mid = Math.floor((start + end) / 2);
            range.setStart(textNode, mid);
            range.setEnd(textNode, end);
            const rects = range.getClientRects();
            if (this.isPointInAnyRect(x, y, rects)) {
                start = mid;
            } else {
                if (x > range.getClientRects()[0].right) {
                    start = end;
                } else {
                    end = mid;
                }
            }
        }

        const caretIndex = start;
        textElement.remove();
        return caretIndex;
    }
    
   createTextOverlay(rect, text, textNode) {
        if (!rect || rect.tagName !== 'rect') return {};

        const textElement = document.createElementNS('http://www.w3.org/2000/svg', 'text');
        const transform = rect.getAttribute('transform') || '';
        const font = rect.getAttribute('data-font-css') || '';

        textElement.setAttribute('x', rect.getAttribute('x'));
        textElement.setAttribute('y', rect.getAttribute('y'));
        textElement.appendChild(textNode);
        textElement.style.setProperty('all', 'initial', 'important');
        textElement.style.setProperty('transform', transform, 'important');
        textElement.style.setProperty('font', font, 'important');
        textElement.style.setProperty('text-anchor', 'start', 'important');

        rect.parentNode.appendChild(textElement);

        const elementRect = rect.getBoundingClientRect();
        const textRect = textElement.getBoundingClientRect();
        const yOffset = ((elementRect.top - textRect.top) + (elementRect.bottom - textRect.bottom)) * 0.5;
        textElement.style.setProperty('transform', `translate(0px,${yOffset}px) ${transform}`, 'important');

        return textElement;
    }

    isPointInAnyRect(x, y, rects) {
        for (const rect of rects) {
            if (x >= Math.floor(rect.left) && x <= Math.floor(rect.right) &&
                y >= Math.floor(rect.top) && y <= Math.floor(rect.bottom)) {
                return true;
            }
        }
        return false;
    }    
            

is this with whitelisting id or works without it as well?

[RafaOstrovskiy]

This is not that easy. But here are some hints. If you have doubt please reach out to me through linkedin.com/in/dbjpanda
I am just pasting some hints. So if you copy paste it might not work. Try to understand the snippets.
Basically you need to create a exact overlay of the rect element and put it on top of that rect element to get the cursor point by enabling the pointer event.

    const cursor = document.querySelector('#kix-current-user-cursor-caret');
    const cursorBbox = cursor.getBoundingClientRect();
    const x = Math.floor(cursorBbox.right);
    const y = Math.floor(cursorBbox.top);
    const rect = this.getRect(x, y);
        
     getRect(x, y) {
        if (!this.styleElement) {
            this.styleElement = document.createElement('style');
            this.styleElement.id = "enable-pointer-events-on-rect";
            this.styleElement.textContent = [
                `.kix-canvas-tile-content{pointer-events:none!important;}`,
                `#kix-current-user-cursor-caret{pointer-events:none!important;}`,
                `.kix-canvas-tile-content svg>g>rect{pointer-events:all!important; stroke-width:7px !important;}`,
            ].join('\n');

            const parent = document.head || document.documentElement;
            if (parent !== null) {
                parent.appendChild(this.styleElement);
            }
        }

        this.styleElement.disabled = false;
        const rect = document.elementFromPoint(x, y);
        this.styleElement.disabled = true;

        return rect;
    }

    getCaretIndex(rect, x, y) {
        const text = rect.getAttribute('aria-label');
        const textNode = document.createTextNode(text);
        const textElement = this.createTextOverlay(rect, text, textNode);

        if (!text || !textElement || !textNode) return null;

        let range = document.createRange();
        let start = 0;
        let end = textNode.nodeValue.length;

        while (end - start > 1) {
            const mid = Math.floor((start + end) / 2);
            range.setStart(textNode, mid);
            range.setEnd(textNode, end);
            const rects = range.getClientRects();
            if (this.isPointInAnyRect(x, y, rects)) {
                start = mid;
            } else {
                if (x > range.getClientRects()[0].right) {
                    start = end;
                } else {
                    end = mid;
                }
            }
        }

        const caretIndex = start;
        textElement.remove();
        return caretIndex;
    }
    
   createTextOverlay(rect, text, textNode) {
        if (!rect || rect.tagName !== 'rect') return {};

        const textElement = document.createElementNS('http://www.w3.org/2000/svg', 'text');
        const transform = rect.getAttribute('transform') || '';
        const font = rect.getAttribute('data-font-css') || '';

        textElement.setAttribute('x', rect.getAttribute('x'));
        textElement.setAttribute('y', rect.getAttribute('y'));
        textElement.appendChild(textNode);
        textElement.style.setProperty('all', 'initial', 'important');
        textElement.style.setProperty('transform', transform, 'important');
        textElement.style.setProperty('font', font, 'important');
        textElement.style.setProperty('text-anchor', 'start', 'important');

        rect.parentNode.appendChild(textElement);

        const elementRect = rect.getBoundingClientRect();
        const textRect = textElement.getBoundingClientRect();
        const yOffset = ((elementRect.top - textRect.top) + (elementRect.bottom - textRect.bottom)) * 0.5;
        textElement.style.setProperty('transform', `translate(0px,${yOffset}px) ${transform}`, 'important');

        return textElement;
    }

    isPointInAnyRect(x, y, rects) {
        for (const rect of rects) {
            if (x >= Math.floor(rect.left) && x <= Math.floor(rect.right) &&
                y >= Math.floor(rect.top) && y <= Math.floor(rect.bottom)) {
                return true;
            }
        }
        return false;
    }    
            

is this with whitelisting id or works without it as well?

I guess it won't, bc it operates with svg annotated from canvas after injecting: window._docs_annotate_canvas_by_ext="WHITELISTED_EXTENSION_ID";

[dbjpanda]

I am thinking to create a package https://github.com/dbjpanda/gdocs-utils. I will push some snippets by EOD.

[ElijZhang]

@RobertJGabriel Hello, could you please tell me how long does it take to get whitelisted? I applied it two days ago, but my extension id is not in whitelist yet.

Mine took around three weeks to be whitelisted 🙈

OK, thanks. (This takes really long ORZ)

[RafaOstrovskiy]

found a way to get the user-selected text or to highlight specific text based on string index.

@ElijZhang Sorry, have you found out a way to get the selected text?

[ElijZhang]

found a way to get the user-selected text or to highlight specific text based on string index.

@ElijZhang Sorry, have you found out a way to get the selected text?

@RafaOstrovskiy I found a not a very good way to get the selected text by analyzing some extension that are in the whitelist.In google docs page which is rendered with _docs_annotate_canvas_by_ext. You may try this code in Chrome devtools:

document.querySelector(".docs-texteventtarget-iframe").contentDocument.execCommand("copy");
const selectedText = document.querySelector(".docs-texteventtarget-iframe").contentDocument.body.innerText

Now you get the selectedText. However, this will change user's clipboard content. I tried to find a way to store user's clipboard content temporarily, and then rewrite it back. But it's hard to do this without the user's awareness.
I would be glad If you found a perfect way and tell me the solution.

[swoorpious]

found a way to get the user-selected text or to highlight specific text based on string index.

@ElijZhang Sorry, have you found out a way to get the selected text?

I have spent the last 3 days reverse engineering Language Tool's code, and it's really easy once you get the initial hold of it.

If there is no selection, you will need to make one. You can check this by it's class (.kix-canvas-tile-selection).

To create a selection, you essentially have to simulate mouse events, which then selects the text in docs.

After making a selection, you send a copy event (new CustomEvent("copy")) to the content editable element in the iframe. Google Docs will then put the selected text in that element which you can get by .innerText.

This does not seem to edit clipboard content in my experience.

[tg44]

@swoorpious or @dbjpanda Can you help me with changing text in the doc? Currently I can get the caret, the selected text, I can list the paragraphs, etc. The only thing I really want to do is "typing" into the document.
I tried;

// focused element
document.activeElement?.dispatchEvent(new KeyboardEvent("keydown", { key: "a" }));
document.activeElement?.dispatchEvent(new KeyboardEvent("keyup", { key: "a" }));
document.activeElement?.dispatchEvent(new KeyboardEvent("keypress", { key: "a" }));

// window just in case
window.dispatchEvent(new KeyboardEvent("keypress", { key: "a" }));

// event target
const b = document.querySelector(".docs-texteventtarget-iframe")?.contentDocument?.body;
  if(b) {
    //console.log(b.querySelector("#docs-texteventtarget-descendant"))
   // with event
    b.querySelector("#docs-texteventtarget-descendant").dispatchEvent(new KeyboardEvent("keypress", { key: "a" }));
   // with forcing
    b.querySelector("#docs-texteventtarget-descendant").innerText = "a";
  }

Seems like none of this worked. What is the trick here?

[luccabb]

@tg44 got write working with the below:

gdocs-enable-annotated-canvas.js

window._docs_annotate_canvas_by_ext = "ogmnaimimemjmbakcfefmnahgdfhfami";

contentScript.js

const writeTextToGDoc = 'hello world'

// write letter to gdoc
function write(letter, doc) {
    const i = new KeyboardEvent("keypress", {
        repeat: !1,
        isComposing: !1,
        bubbles: !0,
        cancelable: !0,
        ctrlKey: !1,
        shiftKey: !1,
        altKey: !1,
        metaKey: !1,
        target: doc,
        currentTarget: doc,
        key: letter,
        code: "Key" + letter.toUpperCase(),
        keyCode: letter.codePointAt(0),
        charCode: letter.codePointAt(0),
        which: letter.codePointAt(0),
        ...{}
    })
    doc.dispatchEvent(i)
}

setTimeout(() => {
    const doc = document.querySelector(".docs-texteventtarget-iframe").contentDocument
    for (const letter of writeTextToGDoc) write(letter, doc)
}, 5000);

manifest.json

...
"content_scripts": [
        {
            "matches": [
                "*://docs.google.com/document/*"
            ],
            "run_at": "document_start",
            "js": [
                "gdocs-enable-annotated-canvas.js"
            ],
            "world": "MAIN"
        },
        {
            "matches": [
                "*://docs.google.com/document/*"
            ],
            "js": [
                "contentScript.js"
            ],
            "all_Frames": false,
            "run_at": "document_end"
        }
    ],
...

[swoorpious]

document.activeElement?.dispatchEvent(new KeyboardEvent("keydown", { key: "a" }));
document.activeElement?.dispatchEvent(new KeyboardEvent("keyup", { key: "a" }));
document.activeElement?.dispatchEvent(new KeyboardEvent("keypress", { key: "a" }));

This should be helpful to anyone looking to type in docs with their code.

@tg44 The way docs handles events is a little unintuitive.

If you want to enter text in the document, you use keypreess events, which specify the code of the character you want to type instead of the character itself. Docs uses code of the character to type, and not the character. Additionally, you will need bubbles: true and cancelable: true for docs to type it.

Note: this enters text at the caret's position

Example code for the above:

static InsertChar (char) {
    const IFTarget = GoogleDocsUtils.GetIFEventTarget();

    for (let i = 0; i < char.length; i++) {
        const eventObj = {
            bubbles: true, // important
            cancelable: true, // important
            key: char,
            keyCode: char.charCodeAt(i), // important
            ctrlKey: false,
            shiftKey: false,
        }

        IFTarget.dispatchEvent(new KeyboardEvent("keypress", eventObj))
    }
    return 1;
}

---

If you want to move the caret with keyboard events, or delete characters (backspace and delete on keyboard), you use keydown events. Again, these will require bubbles: true and cancelable: true for docs to register it.

Example code for the above:

static MoveCaretToLeft (n) {
    const IFTarget = GoogleDocsUtils.GetIFEventTarget();
    const eventObj = {
        bubbles: true, // important
        cancelable: true, // important
        code: "ArrowLeft",
        key: "ArrowLeft",
        keyCode: 37, // important
        ctrlKey: false,
        shiftKey: false,
    }

    for (let i = 0; i < n; i++) {
        IFTarget.dispatchEvent(new KeyboardEvent("keydown", eventObj))
    }

    return 1;
}

---

If you want to select text and move it to other locations in the document, you will need a sequence of mouse and keyboard events. You can try to write them based on how you select text with your mouse. Mouse events need to be dispatched at screen coordinates (pixels) that you can calculate with indexes or caret position.

sorry for not replying in time, i was busy with school haha 😄