ANW-425: Rewrite PUI Collection Organization infinite scroll using native APIs

[brianzelip]

This PR refactors the PUI Collection Organization's mechanism for handling "infinite scrolling", or the just-in-time loading of information related to a (possibly very large) resource.

The bulk of the changes revolves around replacing the JavaScript-based scrolling environment with a native browser-based scrolling environment. At a high level the code changed from "infinite scroll" and "largetree" to "infinite records" and "infinite tree".

The markup and styles were broadly left alone - just tweaks to some html id and data-* attributes and creation of new styles.

The js is what has changed, but it has remained consistent with the general logic of the existing app, producing essentially the same output in the DOM.

The main files of note are the view template and the InfiniteRecords and InfiniteTree class definitions.

The two classes represent similar but different approaches to infinite scrolling given different constraints.

Consistency between this PR and existing code

• on page load
  • no records or tree nodes are avaiable on page load
• after page load:
  • the first two record waypoints of 20 records each are fetched and added to DOM
  • the first waypoint of immediate children of the tree root node are fetched and added to the DOM
  • scroll and click listeners are instantiated
    • "empty" record and tree waypoints get populated as they scroll near their respective scroll viewport
    • when a parent tree node is expanded/collapsed its children are shown/hidden
    • when a tree node is clicked, the record is scrolled into view
  • if there is a url#hash on page load, the waypoint of the hash-specified record and its neighbor waypoints are fetched and populated, and the record is scrolled into view
  • as records are scrolled up and down, the corresponding tree node of the top-most record in the scroll viewport is highlighted as the "current" record

Differences between this PR and existing code

• the current "infinite scroll" and "largetree" code map to this PR's "infinite records" and "infinite tree" code
• the tree no longer scrolls automatically as the records are scrolled (ANW-1737)
• when tree parent nodes are now collapsed, their children are no longer removed from the DOM, but rather stay in the DOM until page reload
• the records section now scrolls on mobile (ANW-276)
• There is now a WIP 'load all remaining records' feature
• A polyfill is included for the scrollend event for Safari and ios.

New use of web APIs and language features

This PR introduces use of web APIs and language features that are new to the ArchivesSpace codebase:

• Intersection Observer - replaces the manual scroll observing that the existing app does 💪
• Web Worker - fetch data off the main thread when loading all remaining records of a large resource
• <template> - native element for templating out markup
• ES6 Classes - the native approach to classes that replaces ArchivesSpace's use of jquery and prototype manipulation
• async/await - leaner asynchronous code that reads synchronously
• generator functions - helpful for handling a high volume of async calls like when loading all remaining records of a large resource
• JSDoc annotations - function description and type annotations added via inline comments
• scrollend event - used for preventing scroll jank when empty record waypoints are observed during programmatic scrolling; a polyfill is included for Safari and iOS, etc.

The "infinite scroll", "largetree", and related code is still used in at least the Collection Overview view, so it cannot be fully removed from the PUI codebase yet.

Further details about the new "infinite records" and "infinite tree"

InfiniteRecords

• Records are presented in a flat, linear view
• In the DOM, records are added in groups called waypoints of WAYPOINT_SIZE length
• There can be any number of waypoints
• All waypoints are siblings of each other
• All records are an immediate child of some waypoint
• The total number of waypoints is known on page load
• Empty waypoints are populated when the user scrolls near them
• Records remain in the DOM until the page is refreshed

InfiniteTree

• Tree nodes are presented in a tree fashion of parent->children groupings
• A node's immediate child nodes are grouped into waypoints of WAYPOINT_SIZE nodes
• Each group keeps track of its nested level count from the root node
• In the DOM, nodes are added either when a parent node is expanded via click, or when an empty waypoint of child nodes is scrolled near
• Any node can have one or more waypoints of children
• The total number of waypoints in the tree is not known on page load
  • however the number of waypoints of child nodes that a parent has is known upon fetching the parent node's data
• Tree nodes remain in the DOM until the page is refreshed

ES6 limitations due to Uglifier build dependency

We experienced CI build failures (via 'ci/dockercloud-stage' which only seems to show up in the GitHub list of checks when it fails) due to the uneven adoption of ES6 syntax by the Uglifier.js dependency which is used for compressing JS assets in production.

• The use of public class fields
• The use of for await...of in the web worker (but potentially not an async generator earlier in the web worker)

Public class field fails

The class field fails were solved by moving the fields to the constructor function.

Web worker for await...of fail

This fail caused a full rewrite of an async generator solution with an ES5 pattern. The ES6 generator code is noted below for provenance (and for reference when Uglifier is replaced!).

onmessage = async e => {
  const { waypointTuples, resourceUri } = e.data;
  const chunks = chunkGenerator(waypointTuples, MAX_FETCHES_PER_PROCESS);

  let done = false;

  for await (let chunk of chunks) {
    postMessage({ data: chunk, done });
  }

  done = true;

  postMessage({ done });

  async function* chunkGenerator(arr, size) {
    for (let i = 0; i < arr.length; i += size) {
      yield await fetchWaypoints(arr.slice(i, i + size), resourceUri);
    }
  }
};

/**
 * fetchWaypoints
 * @description Fetch one or more waypoints of records
 * @param {string} resourceUri - The uri of the collection resource,
 * ie: /repositories/18/resources/11861
 * @param {[wpNum, uris]} waypointTuples - Array of tuples of waypoint metadata
 * @typedef {number} wpNum - The waypoint number
 * @typedef {string[]} uris - Array of record uris belonging to the waypoint
 * @returns {Promise} - A Promise that resolves to an array of waypoint
 * objects, each with the signature: `{ wpNum, records }`
 */
async function fetchWaypoints(waypointTuples, resourceUri) {
  const promises = waypointTuples.map(tuple =>
    fetchWaypoint(...tuple, resourceUri)
  );

  return await Promise.all(promises).catch(err => {
    console.error(err);
  });
}

/**
 * fetchWaypoint
 * @description Fetch a waypoint of records
 * @param {number} wpNum - the waypoint number to fetch
 * @param {string[]} uris - Array of record uris belonging to the waypoint
 * @param {string} resourceUri - The uri of the collection resource,
 * ie: /repositories/18/resources/11861
 * @returns {Promise} - Promise that resolves with the waypoint object made up of
 * keys of record uris and values of record markup
 */
async function fetchWaypoint(wpNum, uris, resourceUri) {
  const origin = self.location.origin;
  const query = new URLSearchParams();

  uris.forEach(uri => {
    query.append('urls[]', uri);
  });

  const url = `${origin}${resourceUri}/infinite/waypoints?${query}`;

  try {
    const response = await fetch(url);
    const records = await response.json();

    return { wpNum, records };
  } catch (err) {
    console.error(err);
  }
}

Tickets this work reflects

• ANW-425 - fix the buggy scrolling
• ANW-276 - stops scrolling on mobile
• ANW-1737 - keep tree static when scrolling records

[donaldjosephsmith]

This appears to work well with my testing, and nothing jumps out at me as being potentially problematic.

[brianzelip]

It should be noted that the commit message for this merged PR is a mess! It was auto-squashed-and-merged before I could clean it up and better present it.