Unable to manage history state programmatically (stream response)

[pySilver]

My use case is fairly simple. I have a list of products and a filter POST form. Whenever a user triggers form submit I receive a stream response that updates the product list and current page URL to something shareable. Filter form itself is contained in a <turbo-frame> if that matters.

My goal is to update the page URL every time user makes changes to a filter form while letting them go back in history and "unapply" filters by simply pressing the Back button in a browser.

Here is my custom stream action:

Turbo.StreamActions.my_push_state = function () {
  const url = this.getAttribute("url");
  history.pushState(history.state, "", url);
  Turbo.navigator.history.push(url);
  console.log("my_push_state", url);
};

And here is the stream response from the server after the filter form is submitted:

<turbo-stream action="replace"><template>...list of products</template></turbo-stream>

<turbo-stream
  action="my_push_state"
  url="http://localhost:3000/?foo=bar"
></turbo-stream>

I cannot make it work. When the Back button is pressed the following error appears in a console:

Uncaught DOMException: Failed to execute 'removeChild' on 'Node': The node to be removed is not a child of this node.
    at ProgressBar.uninstallProgressElement (webpack-internal:///./node_modules/@hotwired/turbo/dist/turbo.es2017-esm.js:1481:38)
    at eval (webpack-internal:///./node_modules/@hotwired/turbo/dist/turbo.es2017-esm.js:1455:22)
uninstallProgressElement @ turbo.es2017-esm.js?7ebb:1457
eval @ turbo.es2017-esm.js?7ebb:1431
setTimeout (async)
fadeProgressElement @ turbo.es2017-esm.js?7ebb:1453
hide @ turbo.es2017-esm.js?7ebb:1430
hideVisitProgressBar @ turbo.es2017-esm.js?7ebb:2070
visitRequestFinished @ turbo.es2017-esm.js?7ebb:2050
finishRequest @ turbo.es2017-esm.js?7ebb:1795
requestFinished @ turbo.es2017-esm.js?7ebb:1920
perform @ turbo.es2017-esm.js?7ebb:529
await in perform (async)
issueRequest @ turbo.es2017-esm.js?7ebb:1767
visitStarted @ turbo.es2017-esm.js?7ebb:2018
start @ turbo.es2017-esm.js?7ebb:1722
startVisit @ turbo.es2017-esm.js?7ebb:2283
historyPoppedToLocationWithRestorationIdentifier @ turbo.es2017-esm.js?7ebb:2931
History.onPopState @ turbo.es2017-esm.js?7ebb:2200

[pySilver]

Ok, the exception comes from the incorrect BrowserSync setup. This config partially solves the issue:

  snippetOptions: {
    rule: {
      match: /<\/head>/i,
      fn: function (snippet, match) {
        return snippet + match;
      },
    },
  },

Credits: https://blog.aidistan.site/2022/10/08/browser-sync-for-turbo-rails.html

However, the problem still partially exists. Now, when I'm pushing the navigation state this way:

Turbo.StreamActions.my_push_state = function () {
  // const url = new URL(this.getAttribute("url"));
  const url = this.getAttribute("url");
  history.pushState(history.state, "", url);
  Turbo.navigator.history.push(url);
  // Turbo.navigator.history.push(url, history.state.turbo.restorationIdentifier);
  // console.log("my_push_state", url);
};

It creates 2 entries in a browser history stack instead of one. When I click Back button once, it does restore the original state but it did not change the URL as expected.

[jaryl]

If you are using Turbo's navigator then you won't have to perform browser's pushState yourself. I imagine it should manage all the current state that Turbo needs correctly as well.

[pySilver]

Well, unfortunately it doesn't work otherwise. I need to dig into sources to see what Turbo.visit does internally, so I can do the same manually:

• update the current URL
• push into history state
• cache new state

If that is not possible I'd have to respond with redirect from my POST form submission then, so Turbo will do all required operations itself. However, that would add additional network round trip that I would like to avoid.

[pySilver]

Apparently, this Redirecting After a Form Submission doesn't do what I expect.

I have a GET form with the action target set as "/explore". (Its browser-sync server, so I cannot really do a POST form)
This /explore endpoint responds with 303 redirect that Turbo follows and page content is being updated, but URL stays the same despite setting data-turbo-action="advance" on a form tag.

I guess I have to make Turbo.visit myself.

[pySilver]

I think I've managed to solve this case by doing the following

Custom Stream Action

Turbo.StreamActions.my_push_state = function () {
  const url = this.getAttribute("url");
  const state = {
    turbo_stream_history: true,
  };
  history.replaceState(state, "", window.location.href);
  history.pushState(state, "", url);
};

What it does is makes sure that the current URL and the new URL are marked for manual visits if navigated back/forward later.

global popstate handler

window.addEventListener("popstate", (event) => {
  if (event.state && event.state.turbo_stream_history) {
    Turbo.visit(window.location.href, { action: "replace" });
  }
});

This handler makes sure we will trigger Turbo.visit for specifically marked URLs.

This approach almost works fine, except it has a cache issue with a page that initiated a stream request (the page is cached in a state right after the stream response is processed) but it gets refreshed thanks to turbo_stream_history marker the first time we visit it again.

[pySilver]

I've managed to accomplish the cache issue by slightly changing custom stream action:

Turbo.StreamActions.my_push_state = function () {
  const url = this.getAttribute("url");
  const state = {
    turbo_stream_history: true,
  };
  Turbo.cache.exemptPageFromPreview();
  history.replaceState(state, "", window.location.href);
  history.pushState(state, "", url);
};

Turbo.cache.exemptPageFromPreview() would mark this page outdated for future previews effectively dropping it from the cache. This can be accomplished without touching internal API I guess, by using:

<meta name="turbo-visit-control" content="reload">

It is also probably possible to simply stop caching in stream response by running Turbo.session.cacheObserver.stop()

[trsteel88]

Turbo.StreamActions.my_push_state = function () {
  const url = this.getAttribute("url");
  const state = {
    turbo_stream_history: true,
  };
  Turbo.cache.exemptPageFromPreview();
  history.replaceState(state, "", window.location.href);
  history.pushState(state, "", url);
};

While this does update the URL, it doesn't seem to store the state in the history. e.g, pressing back in browser history does not restore the previous state.

Nothing this issue is quite old, do you have a solution for this?

[pySilver]

@trsteel88 I left this as-is since in my particular use-case this was a desired behaviour.

[marcoroth]

@trsteel88 in that case you would need to make the "state" part of the actual URL string. You can utilize URLSearchParams for this:

const url = window.location
const params = new URLSearchParams(url.search)

params.set("turbo_stream_history", true)
url.search = params

window.history.pushState({}, "", url);

[trsteel88]

@marcoroth, the URL is updating. However, when navigating back from the browser, the state of the page isn't being replaced to what it was before the URL was changed.

e.g. I'm on /page with a GET form with a query value.

If I type "H", wait, url changes to /page?query=H

Then type, "He", wait, url changes to /page?query=He

When I press back, I expect that form input to change back to "H"

[jeffdlange]

@trsteel88 Trying to get the same behavior you're describing. Were you ever able to get this working?

[jeffdlange]

@trsteel88 Potential solution: render turbo_streams from within a turbo-frame.

Example scenario:
I have a clothes #index page and the user can apply various filters via select-inputs to find what they want (e.g. color, size, etc), but perhaps I also have a sidebar with various categories (e.g. shirts, pants, etc) and clicking on a category should retain any filters I already applied (i.e. If I was searching for medium size blue shirts, clicking on "Pants" in the sidebar should start me off looking at medium, blue, pants). Problem: my main filters and results are in a turbo frame together, whereas my sidebar is in a completely separate div, and will not be updated when the turbo frame updates on form submission.

If we set up our turbo-frame with data-turbo-action="advance" and also render a turbo stream within that frame, we can get the desired behavior:

<%= turbo_frame_tag :products, data: {turbo_action: :advance} do %>
  <%= turbo_stream.replace :sidebar do %>
    <% @categories.each do |category| %>
      <%= link_to category.name, products_path(filters: params[:filters]) %>
    <% end %>
  <% end %>

  <!-- pseudo-code -->
  <%= form_for :filters, method: :get, url: products_path do %>
    <select-input-that-sends params[filters][color]>
    <select-input-that-sends params[filters][size]>

    <%= submit_tag 'Filter' %>
  <% end %>

  <div>
    <!-- products HTML here -->
  </div>
<% end %>

This will:

1. Replace the :products turbo-frame with the response of the form submission (i.e. our newly filtered products)
2. Update the URL to contain the :filters parameters from our form submission
3. Update the sidebar via a turbo stream so that now all our sidebar links will contain our updated filter params

End result is that all our state is captured in the URL, the user does not lose their scroll position, and browser back/forward arrows worked as expected (including working with Turbo's caching, so that you get instantaneous updates when navigating forward/back).

This works because turbo_streams run as soon as they are rendered on the page, so whenever our frame updates, boom, our stream runs as well.

[pySilver]

@jeffdlange that is an interesting solution. So, to be clear, the logic is the following:

1. User submits your faceted filter form
2. Server responds with turbo frame for categories navigation and stream replace command to update categories and faceted form state

?

Can you please provide a sample response from your form?

[jeffdlange]

@pySilver Whoops, I made a mistake in my example. The sidebar links should include an appropriate data-turbo-frame attr look like this:

<!-- sidebar links should target our :products turbo-frame -->
<%= link_to category.name, products_path(filters: params[:filters]), data: {turbo_frame: :products} %>

The above link just hits our ProductsController#index as HTML (although, you could do a more complex controller/view set up where the response includes only the HTML needed for the frame, as opposed to the entire index view, which is what is happening in my example)

[jeffdlange]

@pySilver The request would look something like:

Started GET "/products?filters%5Bcategory_id%5D=1&filters%5Bcolor=blue&size=medium&commit=Filter
Processing by ProductsController#index as HTML

And the response would be our app/views/products/index.html.erb template, containing our :products turbo-frame (and the turbo-frame, in turn, contains the turbo_stream.replace call).

Let me know if that makes sense.

[pySilver]

@trsteel88 did you manage to make it work? I got back to this and yep - it's easy to replace url but it's probably impossible to "cache" new state...

[pySilver]

OK. Doing this directly like this window.Turbo.navigator.history.replace({ href: url }); works fine.

[nfacciolo]

OK. Doing this directly like this window.Turbo.navigator.history.replace({ href: url }); works fine.

Cache is not saved so when we return to the previews page, the page is not reloaded

[krschacht]

It seems like this issue was supposed to be fixed with the addition of data-turbo-action="advance" to turbo-frame, but it's not working as the documentation describes. I've opened an issue for it #1156 and in the meantime I implemented @pySilver 's response above for manually updating the history state. Thanks for that!

[nfacciolo]

It seems like this issue was supposed to be fixed with the addition of data-turbo-action="advance" to turbo-frame, but it's not working as the documentation describes. I've opened an issue for it #1156 and in the meantime I implemented @pySilver 's response above for manually updating the history state. Thanks for that!

I tried data-turbo-action="advance", it works :)

[krschacht]

@nfacciolo Wait, really? Let me state what I tried which isn't working. Maybe you've set it up differently? In essence I have:

<turbo-frame id="conversations-sidebar" target="message">
...
</turbo-frame>

<turbo-frame id="message">
...
</turbo-frame>

When I click a link in the conversations-sidebar, it's output is properly redirected to the message frame. As I watch chrome inspector, the whole <body> does not update, only the turbo-frame updates. However, the browser's URL does not update so the back button does not work.

Then I try adding data-turbo-action="advance" to the conversations-sidebar and now the browser's URL updates and back works, but the whole <body> updates rather than just the turbo-frame and other state on my page is reverted.

I want the frame to update and the URL/browser history to update. Did you get that working? (I only got it working with pySilver's hack for manually updating pushState and Turbo cache, but I'd rather not use this hack)

[jeffdlange]

@krschacht Yes, you aren't the only one with this issue. Once you get into more complex scenarios, especially ones involving the URL and back/forward buttons, Hotwire's tools are not yet robust enough out-of-the-box to handle them (I say this as a huge hotwire fan).

On our project, we have a sidebar that loads via turbo frame, and we want the URL to advance when the sidebar loads (similar to the setup you described in your project). We've only been able to accomplish this via workarounds with custom JS, and it still is not as smooth/polished as we would like.

We've accomplished it by having links-to-the-sidebar literally carrying a url param called sidebar. Then when a user clicks one of those links, we have access to the sidebar URLs in our JS, and can manually do things with Turbo as needed, such as advance the URL (note that even this explanation leaves out some important details, such as managing history).

Please update if you find any simpler solutions, but this it's the best our team has come up with so far.

[krschacht]

Hey everyone, after I confirmed that data-turbo-action="advance" was not working properly in turbo-frames, I had opened an issue just for this. But now I've confirmed in the latest version of turbo rails this has been fixed! It may not address everyone's needs from this thread. It's still not yet programmatic management of history state. But it's an important tool in the toolbox which now works. :)

Details over on this issue: #1156

[dahlbyk]

In #167 (comment) @seanpdoyle said:

I'm not sure whether or not there will ever be support for changing the URL with <turbo-stream> updates...

I wanted to share one straightforward use case where my old team needed history manipulation: renaming the current page. Think GitHub repo:

1. Load https://github.com/owner/old-name/settings
2. Change repository name and save
3. Redirect to https://github.com/owner/new-name

We had a similar situation but wanted to avoid a redirect, so we implemented a custom stream action to history.replaceState() with the updated name. Roughly:

<turbo-stream action="history" target="replaceState">
     <template>https://example.com/thing/new-name</template>
</turbo-stream>

Using <template> is weird, but it worked well enough.

Following from #1240, I would propose instead:

<turbo-stream action="history" method="pushState" url="https://example.com/thing/new-name" />
<turbo-stream action="history" method="replaceState" url="https://example.com/thing/new-name" />

With an optional state attribute to JSON.parse() for the first argument.

It would be trivial to support the other History methods, though their value is unclear.