New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve caching to reduce delay between request and XForms processing #1239

Open
ebruchez opened this Issue Sep 5, 2013 · 2 comments

Comments

Projects
None yet
2 participants
@ebruchez
Collaborator

ebruchez commented Sep 5, 2013

With a form created by a customer, we see about 1000 ms between the logging of the incoming request and the beginning of XForms processing.

Out of this we can see:

  • 220 ms: timing of reading form.xhtml
  • 243 ms: "reading input", which probably does not include the above 220 ms

Some thoughts:

  • XPL processing might take time, but why so long?
  • there should be a conditional GET on the form definition and the processing should be very fast
@ebruchez

This comment has been minimized.

Collaborator

ebruchez commented Jan 16, 2014

How things work now:

  • Form Runner is hooked up in an XPL pipeline
  • new/edit/view page uses model detail-model.xpl and view html-view.xpl
  • pipeline
    • model: read-form.xpl
      • oxf:url-generator
      • XInclude on form definition
      • store in request as fr-form-definition for Summary page
    • view: unroll-form.xpl
      • components.xsl
      • XInclude
    • epilogue.xpl
      • xforms-epilogue.xpl
        • oxf:xforms-to-xhtml ⇒ where static state caching takes place

Caching at the XForms level:

  • see XFormsToSomething
  • cached against processor input
    • Stage1CacheableState (URIReferences)
    • Stage2CacheableState (staticStateDigest + template)
  • XFormsStaticState stored into separate XFormsStaticStateCache, indexed by staticStateDigest

Here is what we expect:

  • XFormsToSomething actual dependencies
    • form definition from persistence layer
      • app name, form name (in URL path)
      • version headers, one of Orbeon-For-Document-Id or Orbeon-Form-Definition-Version
      • NOTE: document id is also passed as document URL parameter.
    • dependent URLs (static instances, schemas, XIncludes)
    • local property files
  • static state must be cached based on those dependencies
  • form definition can also change independently in db (e.g. remote publish) ⇒ ETag

The mechanism must also work with the form version number headers:

  • requires using Vary header
  • use ETag and If-None-Match
  • what ETag could be
    • persistence layer provides it
    • digest of the form definition
    • row id of form definition in database
  • form version made available in the pipeline (which is input to caching too)
    • form-version can be provided as URL parameter to FR page
    • BUT if document id is provided, form-version is ignored and overridden by version returned by persistence layer via orbeon-form-definition-version (or 1 if not found)
  • server can respond with
    • ETag
    • Last-Modified and no Etag
    • none of those
    • Q: should persistence layer be required to have versioning capability?

High-level algorithm proposal:

  • cache keeps entries indexed by app/form name, then ETag
  • probably only keep max of n ETags per app/form name, per LRU
  • incoming request processing
    • request to persistence layer
      • URL w/ app/form name
      • Orbeon-For-Document-Id or Orbeon-Form-Definition-Version header (vary)
      • list of ETags: all ETags in cache for given incoming app/form name
      • must keep ref during request to persistence layer to all cache entries for app/form, so they don't go away
    • response is EITHER 200 + document + ETag
      • create static state
      • cache static state by form/app name then etag
    • OR 304 + one of the ETags which matched (should be only 1)
      • refresh cache entry for app/name/etag
  • persistence implemenatation
    • gets app/form/version headers
    • determines version when needed
    • checks incoming ETags if any
    • responds with 200 + document + Etag or 304 + Etag
    • ETag could be if of document in db (would fail to update if somebody update form def in place, but nobody should!)
  • Q: should cache hold digest, or static state itself (and keep separate static state cache like now)?
    • A: TODO
  • Q: how to handle instance/schema/XInclude dependencies
    • we create the algorithm above within e.g. oxf:form-runner-persistence-client
    • produces key = ETag, validity = 0
    • oxf:xforms-to-xhtml stays connected to its input pipeline as it is now
  • if we use pipeline/custom URL generator, should be equivalent to the above

Next steps:

  • first implementation of oxf:form-runner-persistence-client
  • make sure transformation pipeline is pure
  • document
    • what persistence implementors must do
      • either use ETags (specify what the requirements are for unicity)
      • or use Last-Modified if no versioning required
      • Q: should document URL parameter be removed?

@ebruchez ebruchez added the 4 Points label Apr 15, 2014

@avernet avernet removed this from the Review milestone Oct 17, 2014

@ebruchez ebruchez removed this from the Review milestone Oct 17, 2014

@ebruchez

This comment has been minimized.

Collaborator

ebruchez commented Jun 20, 2017

Following #3272, it would be really good to improve form loading. Requirements:

  • performance
    • conditional GET
    • possibly local caching
  • native Scala
  • testable logic

The ideas in the previous comments still appear sound. In addition, we might try to get rid of the pipeline entirely.

@ebruchez ebruchez changed the title from Long delay between request and XForms processing to Improve caching to reduce delay between request and XForms processing Dec 7, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment