Allow wildcards in @pages for SPA routing, and document

[bob2517]

For generating the @pages list, it could happen that an e-commerce site has a lot of individual product pages:

"/product/1": ...
"/product/2": ...
"/product/3": ...

It's not practical in that case to list them all out in @pages. Instead it needs something like this to handle all cases of a certain type:
"/product/*": ...

For this sort of wildcard handling, page title references, special classes per page, etc would need to come from the link itself or from somewhere else when the page was called, but for something like an e-commerce site the links would usually come from a database of some kind anyway so it's no biggie. We just need the wildcard aspect so that we don't need a gazillion @page entries.

[bob2517]

The only issue with wildcards that I can see from some experimentation, is that wildcard entries details like titles don't get recognised from the back/forward arrows as titles wouldn't be able to have wildcards. The point of the @pages list is to locate URLs. So I can find a URL from a wildcard to see if it matches the link, but if this was from a back or forward arrow, that information is not tied into the history state.

At this point, the best I can do if wildcards are used, is to map the url details to a separate place during runtime. It's not an issue if wildcards aren't used, as all those details could be kept in the @pages list. But wildcards won't cut the cheese in this case, and the data would need to be kept in the link itself, as it's stands at the minute anyway.

But what to do if the back/forward arrows are used after a page refresh? That info is no longer available in memory or from the pages list. That's the only missing link that I can see.

Need a solution for that and then I'll be ready to start the fix for this. Will ponder it and revisit tomorrow. I'm sure there's a simple solution that I'm not seeing yet. Could have a sublist for wildcards, but that defeats the point completely. So not sure yet.

[bob2517]

It's looking like an after event of some kind once the page is loaded. If the data isn't on the page I can't do much more than just load the URL with the wildcard. Not sure beyond that though. I mean, I could use some magic to try and detect url details from URLs on the page, but if redirecting to a fresh page that has no links on it, like a custom 404 page, then there's only so much magic I can do. So it's looking like an after event is needed. Meh - will ponder it some more.

[bob2517]

Maybe an acss special element or something containing all the attributes needed for a url? Just get the developer to have that in dynamically loaded content if using wildcards? Only for a wildcard handling though. Might be the only option. Bit messy though, I don't like it much, but it would work.

[bob2517]

Unless there's something I can do with the history state object. If I could store custom data in there it would work. Hmmm... that gives me an idea - I could use session storage - that might be an option. Now I've got too many options! Which one is the best? WIll come back to this tomorrow, unless I come back to it in a few minutes.

[bob2517]

It's definitely going to be either session-storage or a manipulation of the history object. Definitely. Then the developer doesn't need to get involved. I can just store all the attributes in the link - whether in the @pages list or from the link itself, in the storage place when the link is clicked. Then it will be there for back/forward stuff, etc., and can be gotten from there when wildcards are used with the popstate event - I only need it for the internal popstate event for the SPA magic. Sorted. Will start this tomorrow.

[bob2517]

And it ties seemlessly with what's there already! So, fully backward-compatible upgradableness with no additional syntax required.

[bob2517]

Now I'm half tempted to switch from current template method of SPA routing to a session-storage method. That would be awesome and speed up page loading too. If it gets any faster it's going to be loaded before the person has started up his browser... I dunno the support for session-storage though. It's probably at least the same as the template tag. Will check tomorrow.

[bob2517]

Probably going to use history state for the attributes to handle popstate - it's not actually used in the core and seems the sensible-ist option. Undecided on changing the template approach. Might try some speed tests.

[bob2517]

Might even get some speed improvement on the SPA nav going this route too.

[bob2517]

Now I remember why we need a template. A click is triggered on the URL-linked element which simulates a page switch and runs through the events. Might be a better way to do it.

[bob2517]

Need to list out what's needed here in terms of functionality, as it's a bit complicated and I want to get it out of my head so I can focus on coding it.

1. Don't create the template from @pages as it's done at the moment. Just leave it in memory. Actually, in order to handle the wildcard in the first place we're going to have to do a .find or something like that on an array, so it needs to be in memory anyway - that's an aspect needed, looking at this holistically. Would be quickest on start-up to just store the tag contents as a stringified html tag indexed by a url string, as it won't need any additional formatting. Start-up is the most impacted by changes in this area, but we'll probably see a load speed improvement as we won't need to draw the template that currently contains all the site routing.

2. Person clicks on a link. It's a specific link that has attributes in it that is needed for navigation. It matches a wildcard routing description in @pages, which also contains any additional attributes that is common for that type of link.

So, any new attributes from @pages gets copied over into the link element, and the URL stored in the history state is the URL from the link - not from @pages. Also in the history state is stored a list of the attributes from that specific link. See step 3.

Or is it? What if there's a {$RAND} or something like that as a parameter in the wildcarded url.

I think perhaps the better option is to diff the two URLs and create a larger URL containing any parameters from the link and any parameters from the @pages url.

That should be it for that I think. Can't immediately think of anything that could go wrong with that particularly.

3. On popstate, the url is gotten and the redirect is just done from that. then a click is triggered on the attributes stored in the history state that have been converted to an element and placed into a redirect template tag. Or maybe we can just send a new htmlelement into the event flow? I know I've done recent work on continuing with disconnect elements but I dunno off the top of my head if that'll work or not. Check it out when it comes up.
   [edit] - on the last point, a memory element doesn't work in the event flow. It's not worth changing all the core locations to accept a memory element and not a DOM element, as there could be hidden implications. Will use a template element, then if there are changes to any target selector it'll happen in there.

[bob2517]

The answer to "do we even need the @pages config item or the wildcards in the first place?" is yes. For non-wildcarded items anyway, it means you can build your website just by using hrefs - just hrefs - nothing else - and not having to worry about all the other things that you need to have to get a smooth transition between pages. Like the page title can just be stored once in the @pages list along with the specific url and all transition attributes - and not on several pages of the website html. Really handy if an admin area has something like tinymce and the client will be building links from there.

But wildcarded pages - yes, although common transition attributes can be stored in the @pages list, specific attributes like a data-title attribute for changing page title must be stored in the html - these can't be attached to the wildcard in any way otherwise just by using the @pages list there would also need to be a hundred different page titles with references to a matching url in there too, so there wouldn't be any point. So it's got to be one method or the other or both - wildcards and common transition attributes in the @pages list + link specific attributes in the html, or non-wildcards and common transition attributes and specific attributes in the @pages list.

[bob2517]

Should really do a dedicated docs page explaining how the SPA works. Currently there's a tutorial hidden away, but it should have it's own page explaining how it works.

[bob2517]

Re handling the wildcard aspect of this issue, get the functionality fully rewritten without the wildcards so that it works, then the wildcard handling can be slotted into place, as it's only going to happen in one place - ie. the pre-click location in the core.

[bob2517]

The first step on this is done. So all this works apart from the wildcard bit. In theory it should all just slot into place. It's hard to say if there's been a performance improvement on page load as I was getting 100% performance in lighthouse before. Should have done a comparison beforehand. I did a config load optimization too which would have speeded things up a little bit too. Anyway, the massive template SPA list has been removed from the DOM, which should have made a difference, and it's been replaced with a single element routing div in a template that is needed to get events triggered during the page nav step.

Will work on the wildcard syntax and handling tomorrow. Not finalised what the syntax for the wildcard will look like yet.

[bob2517]

General runtime mouseoverings over links seem a bit faster offline compared to live, which is what I was hoping would happen as the link prep has been speeded up in theory. Could just be my imagination though. It's so fast already it's hard to tell. The back / forward / whatever links in the browser will definitely be faster now, as all the info needed to draw the page is in the history state, so even if the page is refreshed and the browser nav links are used, it stays in an SPA context. I dunno if any other framework even considers getting to this level of awesomeness - it's pretty cool, if I do say so myself.

[bob2517]

In case this isn't clear, this is a backward-compatible upgrade. The only new thing is the performance improvement and the routing wildcards. I'm hoping the wildcard implementation isn't going to take this new performance away too much, but won't know for sure until tomorrow.

[bob2517]

Just writing ideas.

So it will look for a direct match first, always. If it's found then it uses that That's both a performance consideration from me and a stable instruction for developers that would be easy to remember:

@pages {
    "/games/last-of-us": data-ajax="/games/last-of-us" class="pageSwitcher";
}

For direct page referencing - like for dynamic content:

@pages {
    "/games/*": data-ajax="/games/{$1}" class="pageSwitcher";
}

For direct calling of static html inner content that happens to have an extension:

@pages {
    "/games/*": data-ajax="/games/html/{$1}.html" class="pageSwitcher";
}

and

@pages {
    "/games/*/*": data-ajax="/games/{$1}/html/{$2}.html" class="pageSwitcher";
}

Or for older-school people who like extensions everywhere, for server reasons or whatever:

@pages {
    "/games/*.php": data-ajax="/games/{$1}.php" class="pageSwitcher";
}

I could do some sort of regex version, but I'm not sure if that's going to be too complicated for most people trying to write a website. I think that's probably enough. There would be some sort of routing on the server, old-style or new-style, which would handle getting the actual content based on the URL anyway.

[bob2517]

Logic-wise, thinking aloud, it would be a split by slash, comparing array entries - skipping to the next wildcard to check out on failure - when it comes to a wildcard, it gets the character after the * and grabs the content from the lookup up to that character. That content goes into {$1}. Then check the rest of the string if there is anything more. Then continue the check as a pass if all is ok, then repeats if there is another * before the end of the array item, etc. Then once that is all done is goes onto the next array item and continues. If it gets to the end and all matched up then it's a pass for that check and it works out.

Then after that we just regex replace the target attributes in the rest of the wildcard entry with the contents of {$1}, {$2}, etc., everywhere it appears in the attributes, not necessarily in an href url as that could be named anything.

Then after that we can clone the attributes into the element ready for use.

This is why I needed to speed things up in preparation for this - there's a bunch of stuff going on that could slow things up. Unless there's some sort of map compare built-in function that would do this sort of thing already - will check it out. Dunno - I've not written a wildcard thing like this before.

[bob2517]

Need to roll my own for this one.

[bob2517]

Later on, if anyone needs something else other than "*", I can slot that into the core. For now, "*" should be enough for most scenarios I would think. It basically means any series of characters up to the character that follows it in the string to match.

[bob2517]

Would be best to have a nice regex way to do this - could just be a case of matching all the *s up to something, and then comparing everything else to be equal matches, something like that. Dunno, will do something else and come back to this once I've worked it out.

[bob2517]

It could just be a case of converting the wildcard url into a regex on the fly and then just matching it directly with the href. It couldn't be that simple could it? Would just need to capture the wild matching strings to put into replacement variables, but that might work. That would be awesome for performance. Gonna have lunch and ponder it some more.

[bob2517]

That method would also leave it open for other pattern matching other than "*", so it would be easy to add anything else on, if it's needed, without particularly affecting performance.

[bob2517]

Oops - found a bug. The comment delimiters /* and */ are being removed if they are in double-quotes when parsing the config. We don't want that. Fixing that now.

[bob2517]

Comment removal sorted out offline. Will commit when I get this issue finished.

[bob2517]

To expand on what a "*" is, it's not going to be able to contain the characters "/", "\" or ".". I think that should cover sensible dir routings within the URL. So old-school "*.*" will match file + extension and "*" will never match file + extension.

That looks to be needed so that the check doesn't match things it shouldn't. This is almost done I think. May even get a commit today if all goes well.

[bob2517]

Fully functional! Gonna take a break and then optimize it a bit more.

[bob2517]

So this would work to partially SPA an already working config with ajax links that don't already have a @pages declaration if the matter of refreshing a page breaking the SPA flow doesn't matter:

@pages {
	"/books/*": none;
}

or just:

@pages {
	"/books/*":;
}

I'm leaving an option to put "none" in for an empty entry, as it looks a bit nicer in the config. But it's just for aesthetic reasons and doesn't have to be there.

But it's not optimum. The main thing wrong with this approach, is that when a page is refreshed, all the data to get back to it from the browser nav arrows is missing. All the nav attributes, in fact all attributes get stored in the history state object and get fired when the browser nav arrows are used. If all the attributes necessary to switch pages are stored in the link itself, then refreshing a page when not coming from a link, with no attributes to look up in @pages, will end up doing nothing at all. It's a bit confusing, but basically @pages needs to contain nav links and anything necessary for a smooth transition.

Which is why I set up the @pages list in the first place, to hold the attributes necessary for navigation so that it would be possible to SPA a site fully.

It's just an issue for a refreshed page at this point. All the navigation attributes must be in @pages for the full SPA experience to work.

[bob2517]

Actually, thinking about it that "none" for an empty @pages entry shouldn't be there. I'd rather it was stressed that it needs to be filled for it to actually work. So I'll remove that now.

[bob2517]

Meh - the only problem with use wildcards on the docs site, is that there's a lot of attributes that I need for each entry, like ids matching in the menu and stuff like that. There's a lot of jiggery pokery in there for each specific entry. It's just the way it was built, back in the 1700s. Seriously though, I built the version one core from using that website as a test bed, starting with the menu, so it's really old. I'd do the whole thing completely differently if doing it now, but there's no point as I'm going to redo the site later in the year anyway.

All those attributes that reference associated IDs, etc. need to be available on a page refresh for each specific docs page, and unless it's in memory somewhere, that page isn't going to be able to be gotten back to - the menu won't highlight properly, mainly.

I need to make this very clear in the docs: When using wildcards, you can only use it on pages that require a URL, a document title, and only common classes and attributes, in order to fulfil navigation requirements. Essentially, every attribute needed to get back to that page needs to be in @pages. Having specific IDs or attributes needed for a specific page will not work with wildcards. There just isn't anywhere to put them, obviously, as it's a wildcard entry. This is completely different to server-side routing which justs need a URL. On the front-end we need everything available to do the SPA transition.

Hence why it's better just to have one tag with a single href and put everything else into @pages. Then it works like magic.

For an e-commerce site that has a common menu for all products and only requires a change in the url and the title and that needs common transitions for a specific type of product, it'll work fine. But any more than that and it must have individual @pages entries to stay fully immersed in SPA Land.

[bob2517]

Also make it clear that for wildcard entries, all attributes needed for transition need to be matched to the url. Like you can't have different transition attributes for the same wildcard entry, or multiple wildcards with the same URL. It's obvious but it should be mentioned as it can affect how sites need to be built. In that case, if it happened, the differing URLs would just need their own individual entries and not be part of the fallback wildcard. Wildcards are always the fallback option and secondary to an exact match in @pages.

[bob2517]

Even though I can't put wildcards on the docs site, the earlier performance improvements that I made are making mouseovers on links snappier. And I'm still getting 100% in Lighthouse even with the 200 or so docs page listed in @pages. So that's cool.

[bob2517]

Closing for milestone tracking.