Allow online version to read from URL params/hash to set various options + markdown text

[0xdevalias]

I tend to use the online hosted version of this tool a lot:

• https://derlin.github.io/bitdowntoc/

And I was thinking it would be cool if we were able to pass through various options + the markdown text through URL parameters and/or hash. That way I could configure a local shortcut to open the URL with the markdown text pre-filled/etc.

I think there might be limits to the amount of text that can be included in a GET param, so for the markdown text itself, it might make sense to use the hash for that instead maybe?

As an example, something like:

https://derlin.github.io/bitdowntoc/?preset=github&identSpaces=2#markdown=%23%20This%20is%20a%20title%0A%0A%23%23%20Table%20of%20Contents%0A%0A%5BTOC%5D%0A%0A%23%23%20Some%20other%20heading%0A%0ABlah%20blah%20blah.

Which would set the specified options and prefill the following markdown:

# This is a title

## Table of Contents

[TOC]

## Some other heading

Blah blah blah.

[derlin]

Hello @0xdevalias, happy to hear you enjoy this tool 😊 and thank you for this proposal.

There is no difficulty loading options from the hash (I just did a proof of concept). However, I would avoid the markdown text as it will most of the time exceed the maximum length of a URI, and I fear some markdown content may be corrupted when URL-encoded/decoded.

I am also wondering about the design:

1. I already support loading/storing options from the localStorage. Wouldn't both be confusing? I would assume the saved parameters would be loaded first, then the URL parameters applied, if any.
2. should I update the hash (a) every time a parameter changes, (b) only when hitting "generate", or (c) only when the user requests it (with a new button such as copy options to URL)?

I would love to have your opinion on this!

Also, could you please expand on your use case and workflow? Is there a reason not to use the command line?
All the options can be passed as parameters, and thus you could use something like alias bt=bitdowntoc --profile=github ... and call bt README.md from your terminal to generate the TOC.

[derlin]

I made a first proposal that will read the options from the hash on load, and update the hash only upon generate. Would this fit your need? A review is welcome :)

[0xdevalias]

However, I would avoid the markdown text as it will most of the time exceed the maximum length of a URI, and I fear some markdown content may be corrupted when URL-encoded/decoded.

@derlin For my usecase, the markdown content itself is probably the main thing I am interested in; being able to pass through the other settings was a bit of an afterthought.

I'm not 100%, but I believe that other projects may have worked around the 'max length of URL' is specifically by using the hash portion (that isn't sent to the server), rather than the actually GET params. I haven't tested to see if/how large is actually supported in the hash though, so maybe that is a wrong assumption on my part.

---

I am also wondering about the design:

1. I already support loading/storing options from the localStorage. Wouldn't both be confusing? I would assume the saved parameters would be loaded first, then the URL parameters applied, if any.
2. should I update the hash (a) every time a parameter changes, (b) only when hitting "generate", or (c) only when the user requests it (with a new button such as copy options to URL)?

@derlin Hrmm, good question.

For 1, I am sort of 50/50 on it.. on one hand I could see how someone would expect their already saved defaults to be used as the 'base', and then apply the URL params as overrides on top of that; but on another hand I can see how someone might expect the settings passed in the URL to be applied consistently, regardless of what was configured in localstorage. Given that I would sort of expect the people who would end up using URLs with settings in them to be much lower, and the fact that if they really wanted to they could just apply all the settings in the URL, I think I would personally opt for "if settings are present in the URL, then they override the saved localStorage settings for that session, but aren't persisted to localStorage".

For 2, I don't have strong opinions on this one, but given I would expect this to be more of a niche feature, maybe the button to copy options to URL is a good balance of "provide the option for the advanced functionality, but stay out of the way when people don't need it"?

---

Also, could you please expand on your use case and workflow? Is there a reason not to use the command line? All the options can be passed as parameters, and thus you could use something like alias bt=bitdowntoc --profile=github ... and call bt README.md from your terminal to generate the TOC.

@derlin My most recent thoughts/use case that led to this issue was wanting to create an Alfred 'universal action', so that the highlighted/passed through text could be opened in the browser version of this tool:

• https://www.alfredapp.com/universal-actions/
• https://www.alfredapp.com/help/features/universal-actions/

While I haven't thought about it too deeply, I think one of the reasons I was opting for the browser version in this instance is just because that matches my current manual workflow for it.

I think another reason that I was thinking of avoiding the CLI version in this instance was that I wanted to keep the Alfred workflow 'independent of external dependencies', and not need to rely on externally installing the CLI tool, or needing to package it together. Since it's a personal workflow though, I don't know how much either of those really matters in reality.

I could also conceive of usecases where the website supporting it could allow for bookmarklets or similar that could operate in a system independent way; where I wouldn't have the option for using a CLI version; but I admit we're getting into contrived theoretically territory now, and I'm not sure I would practically end up using it much in that way.

---

I made a first proposal that will read the options from the hash on load, and update the hash only upon generate. Would this fit your need? A review is welcome :)

@derlin I left a comment on that PR pointing back to this comment, as I think some of what I provided above sort of answers this question.

[0xdevalias]

This was one tool that I was thinking of that stores settings in the URL hash; though I thought it also saved the 'input' to it, which doesn't seem to be the case:

https://gchq.github.io/CyberChef/#recipe=URL_Encode(false)&oenc=65001

But as a test of length, I took all of the raw markdown from one of my larger gists:

• https://gist.github.com/0xdevalias/d8b743efb82c0e9406fc69da0d6c6581#deobfuscating--unminifying-obfuscated-web-app--javascript-code

URL encoded it with Cyber Chef, which ended up being 228,576 characters long:

And then manually pasted it as a hash to a URL in Chrome, which I then read back from JavaScript with window.location.hash.length, and got the same 228,576 characters.

Though interestingly if I then tried to edit the URL manually, there is a …, and then pressing enter on it again makes window.location.hash.length only return 33,170 characters.

---

Digging a little deeper, I found this:

• https://chromium.googlesource.com/chromium/src/+/master/docs/security/url_display_guidelines/url_display_guidelines.md#url-length

  • In general, the web platform does not have limits on the length of URLs (although 2^31 is a common limit). Chrome limits URLs to a maximum length of 2MB for practical reasons and to avoid causing denial-of-service problems in inter-process communication.

    On most platforms, Chrome’s omnibox limits URL display to 32kB (kMaxURLDisplayChars) although a 1kB limit is used on VR platforms.

https://github.com/chromium/chromium/blob/44ea5cb0d7b1d1a12a8afad0abd661da69f7a83d/content/public/common/content_constants.cc#L17

const size_t kMaxURLDisplayChars = 32 * 1024;

Also this:

• https://stackoverflow.com/questions/812925/what-is-the-maximum-possible-length-of-a-query-string

  • RFC 2616 (Hypertext Transfer Protocol — HTTP/1.1) states there is no limit to the length of a query string (section 3.2.1). RFC 3986 (Uniform Resource Identifier — URI) also states there is no limit, but indicates the hostname is limited to 255 characters because of DNS limitations (section 2.3.3).

    While the specifications do not specify any maximum length, practical limits are imposed by web browser and server software.

    ..snip..

    Microsoft Edge (Browser)
    The limit appears to be around 81578 characters. See URL Length limitation of Microsoft Edge

    Chrome
    It stops displaying the URL after 64k characters, but can serve more than 100k characters. No further testing was done beyond that.

    Firefox (Browser)
    After 65,536 characters, the location bar no longer displays the URL in Windows Firefox 1.5.x. However, longer URLs will work. No further testing was done after 100,000 characters.

    Safari (Browser)
    At least 80,000 characters will work. Testing was not tried beyond that.

    Opera (Browser)
    At least 190,000 characters will work. Stopped testing after 190,000 characters. Opera 9 for Windows continued to display a fully editable, copyable and pasteable URL in the location bar even at 190,000 characters.

    ..snip..

---

and I fear some markdown content may be corrupted when URL-encoded/decoded.

@derlin Yeah, that is a good point. I didn't really think it through all that deeply when I first suggested URL Encoding it; I think some other things I have seen that use it for more 'arbitrary data' like this probably base64 encode it instead.

[derlin]

@0xdevalias Thank you for all this information!

The more I think about it, the more "hacky" it seems. The hash length and how it is processed differently by multiple browsers... What you would need is a real API. However, This is impossible now, as the website is purely static and hosted on GitHub. I just changed the domain so that it may be possible in the future to migrate to something like divio and add new/non-static features.

In the meantime, can you give the CLI a go? (note that an API would be easy to develop with the bitdowntoc CLI: a simple web server calling it in the background and returning the output).

I will see what I do with the PR. This is a nice addition, but I am not sure it will be used and I would like to avoid maintaining "dead" features. And as I understand, you would not use it either, right? (Good points about the loading of the settings though, thanks!)

[0xdevalias]

The more I think about it, the more "hacky" it seems. The hash length and how it is processed differently by multiple browsers... What you would need is a real API. However, This is impossible now, as the website is purely static and hosted on GitHub.

@derlin Yeah, that's fair enough. A real API was definitely what I was hoping for when I first started looking at this.

---

In the meantime, can you give the CLI a go?

@derlin Yeah, I'll try and get it setup in my Alfred workflow when I next get a chance to fiddle with it; and let you know how I go with it.

---

I will see what I do with the PR. This is a nice addition, but I am not sure it will be used and I would like to avoid maintaining "dead" features. And as I understand, you would not use it either, right?

@derlin Totally understandable. Yeah, I think the main part I would be interested in is being able to pass through the markdown itself. While there is variation among browsers, most of them seem to be quite high thresholds; so I wonder if it would end up being much of an issue in reality? Maybe as a 'power user' type feature? I feel like there are sort of less edgecases in supporting the markdown passthrough than even the settings part; as you could essentially just have it decode the markdown from the hash and then add it to the 'paste markdown here' text box; and wouldn't need to worry about if/how that interacts with the local saved settings.

[0xdevalias]

In the meantime, can you give the CLI a go?

Yeah, I'll try and get it setup in my Alfred workflow when I next get a chance to fiddle with it; and let you know how I go with it.

Finally got a chance to look at this again, and created the following workflow in Alfred, using a 'Universal Action':

The > just sets a variable to determine which mode to use later on (based on whether a modifier key is pressed or not)

The first 'Run Script' block calls the bitdowntoc CLI + does some post-processing to change the URL the comment block + removes the inner blank lines:

# Passes the query into bitdowntoc to generate the Markdown Table of Contents, then uses sed to change the URL in the comment + remove the extra lines of spacing

cat <<< "$1" \
| bitdowntoc --profile=GITHUB --indent-spaces=2 --indent-chars=- - \
| gsed -zE '
    # Replace the old URL with the new one
    s|(<!-- TOC start[^>]*)https://github.com/derlin/bitdowntoc([^>]*>)|\1https://bitdowntoc.derlin.ch/\2|g;

    # Remove extra blank line after <!-- TOC start ... -->
    s|(<!-- TOC start[^>]*>)\n\n|\1\n|g;

    # Remove extra blank line before <!-- TOC end ... -->
    s|\n\n(<!-- TOC end[^>]*>)|\n\1|g
'

This is the second 'Run Script' block, which extracts just the Table of Contents from the output:

cat <<< "$1" | sed -n '/<!-- TOC start/,/<!-- TOC end/p'

It could be cool if the CLI supported a 'Table of Contents only' output mode, which would remove the need for the second post-processing script; but all in all, this seems like it should work for my needs pretty well.

[derlin]

@0xdevalias bitdowntoc now supports the TOC only option 😊. Already live on the Web, will be on the cli in the next release. Let me know what you think a(nd if we can close this issue).

[0xdevalias]

@derlin Checked the web version and it seems to work well. If I was nit picking I'd say the style of the button doesn't stand out great, but aside from that, all good.

Let me know what you think a(nd if we can close this issue).

Given earlier discussions around how the full markdown in URL hash seems infeasible, that it's not possible to have an API currently, and that the CLI version + alfred workflow has solved my original needs/want for asking for this, I think it's probably fine to close this issue as wontfix or similar now :)