Stopping state from growing forever (or: limiting displayed state)

[command-tab]

I'm making really great progress having taken the real-world example and gotten it working with my own app's remote data, including session information, listing several collections, and displaying detail views for items within those collections. I couldn't be more stoked to see this all in action! However, I'm running into an issue, and I'm wondering what the recommended design approach is.

As I browse around my app, my entities portion of my state grows, gathering up things that were, at one time, fetched and displayed in the app. When I return to a list view, there are more entities displayed than when I started.

For a concrete GitHub-based example, say my app can list and detail users and repos. If I show a list of users and see Alice and Bob, then look at the detail view for a repo owned by Mallory, then re-visit my user list, suddenly I see Alice, Bob, and now Mallory. Because I had to load information about Mallory in the past, now she's remains in the present state.

Should I:

• Store only the most recent result output from normalizr, and only display those referenced entities? (i.e. Only show the last set of data we're sure we received from the back end. Perhaps I should be doing this anyway for ordering's sake — are there any examples? Edit: Duh self, right here)
• Use redux-react-router to dispatch a Redux action upon route change which (via a reducer) wipes out my state.entities in the store?
• Dispatch a Redux action upon componentDidMount() which (via a reducer) wipes out my state.entities in the store?
• Some other approach?

Thanks again for this amazing project :)

[gaearon]

Good question.
Are you sure it's really an issue though?
Do users spend hours in your app without closing, all the while consuming different data?
Are they looking at thousands of records?

This may be a premature optimization.

[gaearon]

When I return to a list view, there are more entities displayed than when I started.

Why is that? You shouldn't enumerate entities directly—they are just a cache.
You should primarily use a list of IDs (which can be paginated, sorted, etc).

Store only the most recent result output from normalizr, and only display those referenced entities? (i.e. Only show the last set of data we're sure we received from the back end. Perhaps I should be doing this anyway for ordering's sake — are there any examples? Edit: Duh self, right here)

Yes.

[command-tab]

I'll give that a shot — it seems like the way to go.

Thanks so much!

[wmertens]

You could also tag entities with a timestamp and have the reducer remove
old entries - completely stateless if you include time into your state ;)

On Fri, Aug 28, 2015 at 12:14 AM Collin Allen notifications@github.com
wrote:

Closed #644 #644.

—
Reply to this email directly or view it on GitHub
#644 (comment).

Wout.
(typed on mobile, excuse terseness)

[gaearon]

One caveat is that your newer entities might reference (by IDs) the older entities in cache.
You don't want to break your references.

And the only way to be cautious about it is to make schema first-class (like it is in normalizr), and generate the reducer from it, so that it knows whether some entity is unreferenced from any ID list or any other entity. Like a garbage collection.

Again, I don't suggest you to write it unless you're sure it's a problem.

[behl1]

@gaearon Although this issue is closed, I have a use case where I am a bit lost related to this. Say I have a messaging app. I got a list of messages, which I properly recorded in my entities and cached their ids in my pagination state. Now, I write a message and send to the server. I update my entities and ids accordingly, but can only use something like a client_id to save this for now, as I do not have a success reply from my server. How do I remove this temporary message after I receive a success from my server to accommodate the real id of the message and future logic related to that id? Sorry for the trouble and thanks for your time

[gaearon]

Fire an action with both IDs and handle it in the reducer? The exact tradeoffs really depend on what you do in your app. There’s no magically correct approach here—consider different tradeoffs and choose the ones you prefer 😉

[behl1]

@Gaeron True. Sorry for the trouble :)

[atticoos]

I'm definitely in the camp of look for a garbage collector. I work on a tablet app that is mounted outside conference rooms running 24/7 that consumes and displays data relative to the current day. Whenever the day changes, the previous day's data is no longer needed. When a week, month, etc goes by, this can become a problem.

I'm thinking about having an interval, say every 6 hours, run through and GC the entities that no longer have references. This can also happen when certain screens unmount since there's some screens that load data that no other screens care about. However, the "main" screen is running 90% of the time and this rarely unmounts.

I'd love to create a "reference counter" reducer to track entity references and make it easy to remove entities based on the 0 reference counts. However, I may just do it manually, where I can scan for known ID lists/pointers in other reducers and remove what's no longer referenced in the entities reducer. Having a "reference counter" reducer would be nice though to immediately know what's no longer needed. Maybe that could even be a selector..

[atticoos]

Perhaps something like

Assume there a, b, c are entity types

// selector
const knownEntityIdListSelectorA = state => state.screenA.someEntityList;
const knownEntityIdListSelectorB = state => state.screenB.someEntityList;
const knownEntityIdListSelectorC = state => state.screenC.someEntityList;
const garbageCollectorSelector = createSelector(
  knownEntityIdListSelectorA,
  knownEntityIdListSelectorB,
  knownEntityIdListSelectorC,
  entitiesSelector,
  (listA, listB, listC, entities) => ({
     a: Object.keys(entities.a).filter(a => listA.indexOf(a) === -1),
     b: Object.keys(entities.b).filter(b => listB.indexOf(b) === -1),
     c: Object.keys(entities.c).filter(c => listC.indexOf(c) === -1)
  })
)

// action
function garbageCollectorActionCreator() {
  return (dispatch, getState) => {
    var entities = garbageCollectorSelector(getState());
    dispatch({
      type: ENTITY_GARBAGE_COLLECT,
      entities
    });
  }
}

// root component
componentDidMount() {
  this.gcInterval = setInterval(
    () => store.dispatch(garbageCollectorActionCreator(), 
    6 * 60 * 60 * 1000 // 6 hrs
  )
}

And the entities reducer would handle purging upon the action ENTITY_GARBAGE_COLLECT.

My only complaint here is that this is very maintained.

It'd be nice if it could be a bit more implicit with a "reference counter" reducer that would always track references based on the actions throughout the application, and that could be used to capture what should be removed at any given time (based upon zero ref counts)

[atticoos]

As an alternative idea, a higher order reducer may very well be useful here.

[johnsoftek]

If you don't hold references to the old data, it will be garbage collected
by JS engine.

Sent from my phone
On Mar 8, 2016 1:24 AM, "Atticus White" notifications@github.com wrote:

As an alternative idea, a higher order reducer
http://redux.js.org/docs/recipes/ImplementingUndoHistory.html#meet-reducer-enhancers
may very well be useful here.

—
Reply to this email directly or view it on GitHub
#644 (comment).

[atticoos]

Yes, but we're talking about the state in the application. That's kept in reference by the store -- by "garbage collection" we're not talking about the same ideas as the JS engine. This is looking at how to prune state when things are no longer needed.

[gaearon]

Relevant example: https://github.com/reactjs/redux/pull/1275/files#diff-c0354bd716170b207154b564fb9a209aR58
In this case “collection” happens immediately but one can imagine this happening later.

[johnsoftek]

@ajwhite Why are you keeping the reference to old data in the store if the app does not need it?

[johnsoftek]

@ajwhite Ah, maybe you are talking about old model data rather than view model.

[atticoos]

@johnsoftek this isn't really what I'm talking about. I'm dealing with an entity cache with a reducer that stores data models transformed by normalizer. This doesn't really have anything to do with traditional JavaScript variable references. By "references" I'm referring to entity IDs in other reducers that refer to objects in the entity reducer.

[wmertens]

Add a timestamp in the data coming from the actioncreators and in the
reducer remove cached data with timestamps older than x time from the
timestamp.

Alternatively, only keep x items in the reducer and delete the oldest first.

The reducers remain pure.

On Tue, Mar 8, 2016, 1:06 AM Atticus White notifications@github.com wrote:

@johnsoftek https://github.com/johnsoftek this isn't really what I'm
talking about. I'm dealing with an entity cache with a reducer that stores
data models transformed by normalizer. This doesn't really have anything to
do with traditional JavaScript variable references. By "references" I'm
referring to entity IDs in other reducers that refer to objects in the
entity reducer.

—
Reply to this email directly or view it on GitHub
#644 (comment).

Wout.
(typed on mobile, excuse terseness)

[atticoos]

Add a timestamp in the data coming from the actioncreators and in the
reducer remove cached data with timestamps older than x time from the
timestamp.

I think I'd still need to ensure there's no references from other reducers, otherwise data could be dropping while expected to be there.

Alternatively, only keep x items in the reducer and delete the oldest first.

Would want to avoid capping off at a certain size, versus just removing what's no longer needed

[atticoos]

I've been brainstorming some ideas on how to count references that other reducers have to a given entity. This is all just a psuedo example, so it's only to be thought of at a higher level. I'm sure there's implementation details in here that could be done better.

The idea is a reducer enhancer can be used by taking the entities reducer, and the rest of the reducers, and combine them and observe when new references are made or old ones are removed.

I've wanted to avoid:

• Removing after a certain amount of time, as it still matters if anything has a reference
• Capping the entities to a certain size

This still has a risk of:

• Less explicit relationships. This currently relies on an ID reference from another reducer. It's still very possible that relationships are made in different ways at the selector level, making it much tougher to determine what entities are going to be referenced.

Creating the reducers

This will enhance the reducers with reference counting logic

import referenceCountEnhancer from './referenceCountEnhancer';
import entityReducer from './entities';
import reducerA from './reducerA';
import reducerB from './reducerB';
import reducerC from './reducerC';
//...

// combines all reducers but also manages reference counts
export default referenceCountEnhancer(entityReducer, {
  reducerA,
  reducerB,
  reducerC
});

A reducer that references entities

A reducer that produces state with references to entities will contain fields that describe the references that they point to

const initialState = {
  foo: 'bar',
  foobar: 'foobar',
  things: [],
  otherThing: null
}

// this will add some attributes to the reducer that the enhancer will use to determine references
@References({
  things: Entities.Things,  // normalizr Schema used by entity reducer
  otherThing: Entities.OtherThings  // normalizr Schema used by entity reducer
})
export function reducerA(state = initialState, action) {
  //...
}

The reference counting enhancer

This will take the reducers that make references to the entities reducer, determine the references during a state change, and trim the entities state if any references become dropped.

function referenceCountsFromReducers(entityReducer, reducers, state, action) {
  // determine references from each reducer to the entities
  // end up with:
  // {
  //   things: {
  //     reducerA: [..ids referenced...],
  //     reducerB: []
  //   },
  //   otherThing: {
  //      reducerA: [...ids referenced...],
  //      reducerB: []
  //   }
  // }
  //
}

// reduces the entities to only entities that have references, omitting those that don't
function entitiesWithReferences(entities, references) {
  const onlyRefs (entity, refs) => {
    return Object.keys(entities[entity])
      .filter(id => refs.indexOf(id) > -1)
      .reduce((reducedEntity, id) => ({
        ...reducedEntity,
        [id]: entities[id]
      });
  }

  return Object.keys(entities).reduce((reducedEntities, entity) => {
    var refs = Object.keys(references[entity]).reduce((acc, refs) => acc.concat(refs), []);
    return {
      ...reducedEntities,
      entity: onlyRefs(entities[entity], refs)
    }
  });
}

export function referenceCounterEnhancer (entityReducer, referenceableReducers) {
  const combinedReducers = combineReducers({
    entities: entityReducer,
    ...referenceableReducers
  });
  const initialState = {
    ...combinedReducers(undefined, {}),
    referenceCounts: referenceCountsFromReducers(entityReducer, referenceableReducers)
  };

  return function (state = initialState, action) {
    let next = combinedReducers(state, action);
    let entities = next.entities;

    // compares reducers results, ignores `referenceCounts` as thats part of the enhancer
    if (!stateHasChanged(next, state)) {
      return state;
    }

    // find the references for the reducers
    let referenceCounts = referenceCountsFromReducers(next);
    if (referenceCounts !== state.referenceCounts) {
      // remove any no longer referenced entities
      next.entities = entitiesWithReferences(referenceCounts, entities);
    }
    return next;
  };
}

[olalonde]

@ajwhite even if you figured out an elegant way to determine whether an entity is being referenced in other parts of the state tree, you still wouldn't know if it is safe to "garbage collect" the entity because some random component might assume that this entity will be available in the state, right?

Or do you have some kind of rule in place that in order to use an entity in a component, it must be referenced to at least in one other place in the state tree (so that you can safely assume non referenced entities are not required for any up coming component render)?

I've been thinking about possible solutions to the redux state garbage collection problem a bit and I can almost always find cases where an automatic garbage collector would break. But I don't think it's totally hopeless :).