Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

XPath: replace reflective calls with native functions #2214

Closed
ebruchez opened this issue May 7, 2015 · 16 comments
Closed

XPath: replace reflective calls with native functions #2214

ebruchez opened this issue May 7, 2015 · 16 comments

Comments

@ebruchez
Copy link
Collaborator

ebruchez commented May 7, 2015

Scala.js doesn't have reflection, so we cannot use direct Java/Scala calls. We could imagine using JSExport in Scala.js to expose functions, but we won't be able to dynamically know their parameter types, which is needed right now to convert parameters back and forth.

There are about 70 functions used by Form Runner. They are gathered below. For reference, I added also the Form Builder functions, which we might want to expose natively on the server too.

Only a subset of those might make sense on the client.

Steps:

  • create internal use Saxon function library/libraries
  • Q: could we use a Scala macro to generate Saxon functions?

Functions in org.orbeon.oxf.fr.FormRunner:

  • frf:allAuthorizedOperationsAssumingOwnerGroupMember
  • frf:authorizedOperationsBasedOnRoles
  • frf:autosaveSupported
  • frf:buildPDFFieldNameFromHTML
  • frf:canNavigateNew
  • frf:canNavigateSummary
  • frf:canPublishLocal
  • frf:canPublishLocalToRemote
  • frf:canPublishRemote
  • frf:canPublishRemoteToLocal
  • frf:canSelectLocalNewer
  • frf:canSelectPublishedLocal
  • frf:canSelectPublishedRemote
  • frf:canSelectRemoteNewer
  • frf:canSelectUnpublishedLocal
  • frf:canSelectUnpublishedRemote
  • frf:canUnpublishLocal
  • frf:canUnpublishRemote
  • frf:canUpgradeLocal
  • frf:canUpgradeRemote
  • frf:collectDataAttachmentNodesJava
  • frf:controlNameFromId
  • frf:controlNameFromIdOpt
  • frf:controlSortString
  • frf:createFormDataBasePath
  • frf:createFormDefinitionBasePath
  • frf:createFormMetadataDocument
  • frf:currentFormResources
  • frf:duplicate
  • frf:errorMessage
  • frf:findRepeatedControlsForTarget
  • frf:getFormLangSelection
  • frf:getPDFFormatExpression
  • frf:getPDFFormats
  • frf:htmlFieldLabel
  • frf:hyperlinkURLs
  • frf:isLocalNewer
  • frf:isLocalUnavailable
  • frf:isMultipleSelectionControl
  • frf:isRemoteAvailable
  • frf:isRemoteNewer
  • frf:isRemoteUnavailable
  • frf:isSingleSelectionControl
  • frf:isWizardValidate
  • frf:joinLocalAndRemoteMetadata
  • frf:orbeonRolesSequence
  • frf:ownerGroupPermissionsSupported
  • frf:pdfFilenameOrNull
  • frf:publish
  • frf:remoteServersXPath
  • frf:resolveTargetRelativeToActionSource
  • frf:selectFormLang
  • frf:selectFormRunnerLang
  • frf:sendError
  • frf:showCaptcha
  • frf:successMessage
  • frf:topLevelSectionNameForControlId
  • frf:topLevelSectionsWithErrors
  • frf:updateIteration
  • frf:versioningSupported
  • frf:xpathAllAuthorizedOperations

Functions in org.orbeon.oxf.fr.process.SimpleProcess:

  • process:runProcess
  • process:runProcessByName

Other functions:

  • org.orbeon.oxf.fr.DataMigration
    • migration:dataMaybeMigratedFrom
    • migration:dataMaybeMigratedTo
    • migration:findLegacyGridBindsAndTemplates (FB)
    • migration:buildGridMigrationMap (FB)
  • org.orbeon.oxf.util.SecureUtils
    • secure:randomHexId
  • org.orbeon.oxf.util.ScalaUtils
    • utils:truncateWithEllipsis
  • org.orbeon.oxf.common.Version
    • version:isPE

Functions in org.orbeon.oxf.fb.FormBuilder (won't be used on client):

  • fbf:MinimalIEVersion
  • fbf:bindId
  • fbf:bindingForControlElementOrEmpty
  • fbf:buildFormBuilderControlAbsoluteIdOrEmpty
  • fbf:buildFormBuilderControlEffectiveIdOrEmpty
  • fbf:containerById
  • fbf:controlNameFromId
  • fbf:controlsInCol
  • fbf:controlsInContainer
  • fbf:controlsInRow
  • fbf:countAllControls
  • fbf:countAllNonContainers
  • fbf:countGrids
  • fbf:countRepeats
  • fbf:countSectionTemplates
  • fbf:countSections
  • fbf:currentLang
  • fbf:defaultIterationName
  • fbf:deleteCellContent
  • fbf:deleteCol
  • fbf:deleteGridById
  • fbf:deleteRow
  • fbf:deleteSectionById
  • fbf:deleteSectionTemplateContentHolders
  • fbf:expandCell
  • fbf:expandCellTouchesControl
  • fbf:findBindByNameOrEmpty
  • fbf:findBlankLHHAHoldersAndElements
  • fbf:findControlBoundNodeByName
  • fbf:findControlByNameOrEmpty
  • fbf:findRepeatIterationNameOrEmpty
  • fbf:findSchemaOrEmpty
  • fbf:findSchemaPrefixOrEmpty
  • fbf:formBuilderPermissionsAsXML
  • fbf:getAllControlNamesXPath
  • fbf:getBindNameOrEmpty
  • fbf:getContainerNameOrEmpty
  • fbf:getControlHelpOrEmpty
  • fbf:getControlItemsGroupedByValue
  • fbf:getFormDoc
  • fbf:getNormalizedMaxOrEmpty
  • fbf:getNormalizedMin
  • fbf:gridCanDoClasses
  • fbf:hasCustomIterationName
  • fbf:hasEditor
  • fbf:initializeGrids
  • fbf:insertColLeft
  • fbf:insertColRight
  • fbf:insertRowAbove
  • fbf:insertRowBelow
  • fbf:isBrowserSupported
  • fbf:isContentRepeat
  • fbf:isControlLHHAHTMLMediatype
  • fbf:isLegacyRepeat
  • fbf:isRepeat
  • fbf:iterateSelfAndDescendantBindsResourceHoldersXPath
  • fbf:metadataInstanceRoot
  • fbf:moveSectionDown
  • fbf:moveSectionLeft
  • fbf:moveSectionRight
  • fbf:moveSectionUp
  • fbf:nextIds
  • fbf:possibleAppearancesByControlNameAsXML
  • fbf:publish
  • fbf:readDefaultAlertAsXML
  • fbf:readMipAsAttributeOnlyOrEmpty
  • fbf:readValidationsAsXML
  • fbf:renameControlIfNeeded
  • fbf:renameControlIterationIfNeeded
  • fbf:resourcesInstanceRoot
  • fbf:resourcesRoot
  • fbf:sectionCanDoClasses
  • fbf:setControlHelp
  • fbf:setControlItems
  • fbf:setControlLHHAMediatype
  • fbf:setRepeatProperties
  • fbf:shrinkCell
  • fbf:updateMipAsAttributeOnly
  • fbf:writeAlertsAndValidationsAsXML

Other functions which won't be used on client:

  • misc
    • java.util.Map
    • org.orbeon.oxf.fb.ToolboxOps
    • org.orbeon.oxf.fb.MigrationOps
    • org.orbeon.oxf.pipeline.api.FunctionLibrary
    • org.orbeon.oxf.processor.pipeline.PipelineFunctionLibrary
    • org.orbeon.oxf.util.NetUtils
    • org.orbeon.oxf.xml.SaxonUtils
    • org.orbeon.oxf.xforms.processor.XFormsResourceServer
  • simple-captcha.xbl
    • org.orbeon.xbl.SimpleCaptcha
  • evaluate.xsl
    • org.orbeon.saxon.sxpath.IndependentContext
    • org.orbeon.saxon.sxpath.XPathEvaluator
    • org.orbeon.saxon.sxpath.XPathExpression
@ebruchez ebruchez self-assigned this May 7, 2015
@ebruchez ebruchez changed the title XPath: replace reflection calls with native functions XPath: replace reflective calls with native functions May 7, 2015
@ebruchez
Copy link
Collaborator Author

Scala macro annotations should be able to handle this. You would write something like:

@XPathFunction
def runProcessByName(scope: String, name: String): Unit = ...

and this would produce:

class RunProcessByNameFn extends XFormsFunction with FunctionSupport {
    override def iterate(xpathContext: XPathContext): SequenceIterator = {
        // 1. Convert function parameters
        ...
        // 2. Call native function
        val result = runProcessByName(...)
        // 3. Convert and return function result
        ...
    }
}

Type conversions, currently done in XPath.scala and JavaExtensionLibrary.java, would have to be done at compile time.

The functions created this way must also be registered with a function library, with the proper XPath signature:

Fun("run-process-by-name", classOf[RunProcessByNameFn],  ...)

Open questions:

  • How many function libraries should be created this way? One per class with @XPathFunction? Is this possible?
  • How to specify a namespace for that function library?

@ebruchez
Copy link
Collaborator Author

ebruchez commented Jun 1, 2015

As a first step, we should place an @XPathFunction annotation on all functions which are effectively called from XPath. This will help for clarity, and also can later be used for that macro if we implement it.

ebruchez added a commit that referenced this issue Jun 10, 2015
- missing: Form Builder functions
- missing: Other functions which won't be used on client
- still need to create an actual annotation
ebruchez added a commit that referenced this issue Jul 13, 2015
- must still go through all FB functions, see #2214
@avernet
Copy link
Collaborator

avernet commented Feb 4, 2016

[+1 from community] for process:runProcess()

@ebruchez
Copy link
Collaborator Author

Some pointers for Scala macro annotations:

@ebruchez
Copy link
Collaborator Author

ebruchez commented Sep 26, 2016

Tasks:

  • place an //@XPathFunction comment on all functions which are effectively called from XPath [mostly done, will have to double-check when we actually implement]
  • fairly easy: implement dummy macro annotation
    • sbt setup
    • add annotation implementation
    • move //@XPathFunction to actual annotation @XPathFunction
    • get dummy annotation to output information during build
  • actual implementation
    • generate Saxon signatures
    • generate Saxon stubs which call Scala functions (assuming we keep original functions)
    • figure out how to actually create function library, maybe just a call in the enclosing class/trait which registers the function?

@ebruchez
Copy link
Collaborator Author

ebruchez commented Sep 26, 2016

Updated example of generated code:

@XPathFunction
def runProcessByName(scope: String, name: String): Unit = ...

MyLibrary.registerFunction(
  // Pass signature
  xpathName   = "run-process-by-name",
  minArity    = 2, // needed?
  returnType  = Type.ITEM_TYPE, EMPTY,
  args        = List(
    Arg(STRING, EXACTLY_ONE),
    Arg(STRING, EXACTLY_ONE)
  ),
  constructor = new XFormsFunction with FunctionSupport {
    override def iterate(xpathContext: XPathContext): SequenceIterator = {
        // 1. Convert function parameters
        ...
        // 2. Call native function
        val result = runProcessByName(...)
        // 3. Convert and return function result
        ...
    }
  }
)

@ebruchez
Copy link
Collaborator Author

Just hit a bug whereby we incorrectly handle, at runtime, a returned Seq[String] and need instead to use a SequenceIterator. Knowing types at compile-time would avoid this kind of issues.

@ebruchez
Copy link
Collaborator Author

We now group all Form Builder functions in FormBuilderXPathApi and ToolboxOps. We should do the same for Form Runner functions.

@ebruchez
Copy link
Collaborator Author

ebruchez commented Jun 9, 2019

The Scala side of this feature falls under the "macro paradise" plugin. As of Scala 2.13, this is still experimental, but it is now part of the compiler (see also release notes).

@ebruchez
Copy link
Collaborator Author

ebruchez commented Dec 2, 2020

Q: Do we create one function library per file, or do we have the implementation in the file register the functions to another, external library?

The annotation could have parameters if needed:

@XPathFunction(ns = "...", lilbrary = ..., xmlName = "...")

As a start, make the enclosing class/object explicitly a function library. The macro generates registration code for that.

@ebruchez
Copy link
Collaborator Author

ebruchez commented Dec 2, 2020

Classes/objects that expose @XPathFunction:

  • Version.scala (3)
  • FormBuilderMigrationXPathApi.scala (3)
  • FormBuilderXPathApi.scala (66)
  • ToolboxOps.scala (10)
  • WorkflowUI.scala (23)
  • FormBuilderPermissionsOps.scala (2)
  • FormRunnerActionsOps.scala (12)
  • FormRunnerBaseOps.scala (12)
  • FormRunnerContainerOps.scala (3)
  • FormRunnerControlOps.scala (5)
  • FormRunnerEmail.scala (4)
  • FormRunnerHome.scala (25)
  • FormRunnerImport.scala (3)
  • FormRunnerLang.scala (5)
  • FormRunnerMetadata.scala (2)
  • FormRunnerPDF.scala (4)
  • FormRunnerPermissionsOps.scala (5)
  • FormRunnerPersistence.scala (11)
  • FormRunnerPublish.scala (1)
  • FormRunnerResourcesOps.scala (2)
  • FormRunnerSectionTemplateOps.scala (1)
  • FormRunnerSummary.scala (2)
  • GridDataMigration.scala (4)
  • NodeInfoCell.scala (2)
  • SimpleDataMigration.scala (2)
  • Permissions.scala (1)
  • FormRunnerRenderedFormat.scala (1)
  • ProcessInterpreter.scala (2)
  • ConverterTest.scala (3)
  • ContentTypes.scala (1)
  • SecureUtils.scala (1)
  • StringUtils.scala (1)
  • XFormsUtils.scala (2)
  • XFormsUploadControl.scala (1)
  • SaxonUtils.scala (1)
  • SaxonUtils.scala (1)
  • DateSupportJava.scala (2)
  • ErrorSummary.scala (2)
  • NumberSupport.scala (4)
  • Wizard.scala (5)
  • RouterBase.scala (1)

@ebruchez
Copy link
Collaborator Author

ebruchez commented Dec 2, 2020

We can also consider explicitly or implicitly passing, depending on the function signature:

  • XPathContext
  • XFormsFunction.Context

For example:

@XPathFunction
def foo(
  a:    String,
  b:    Option[om.NodeInfo])(implicit
  xpc:  XPathContext,
  xffc: XFormsFunction.Context
): Iterable[om.NodeInfo] = ???

In that case, the macro would pass the parameters.

@ebruchez
Copy link
Collaborator Author

ebruchez commented Dec 3, 2020

We now have the first simple functions implemented with macros on the offline branch. Missing:

  • Iterable[] return param/type
  • XPathContext and XFormsFunction.Context implicits (or explicits)
  • function flags (focus, etc.)
  • migrate all XForms functions
  • migrate/annotate all Form Runner functions
  • migrate/annotate all Form Builder functions

@ebruchez
Copy link
Collaborator Author

ebruchez commented Dec 9, 2020

For a small Form Builder-built form, here are the functions used in the resulting XForms:

  • ErrorSummary:removeUpdateOrInsertError
  • ErrorSummary:updateForMovedIteration
  • frf:allAuthorizedOperationsAssumingOwnerGroupMember
  • frf:authorizedOperationsBasedOnRolesXPath
  • frf:clearMissingUnsavedDataAttachmentReturnFilenamesJava
  • frf:createNewFromServiceUrlOrEmpty
  • frf:getFormLangSelection
  • frf:isAutosaveSupported
  • frf:isLeaseSupported
  • frf:selectFormLang
  • frf:selectFormRunnerLang
  • frf:sendError
  • frf:updateTemplateFromInScopeItemsetMaps
  • frf:xpathAllAuthorizedOperations
  • frf:xpathFormRunnerStringProperty
  • frf:xpathOrbeonRolesFromCurrentRequest
  • frp:providerDataFormatVersion
  • grid-migration:dataMaybeMigratedFromDatabaseFormat
  • grid-migration:dataMaybeMigratedFromEdge
  • grid-migration:dataMigratedToEdgeOrEmpty
  • metadata:createFormMetadataDocument
  • secure:randomHexId
  • simple-migration:dataMaybeWithSimpleMigration

We can prioritize those.

@ebruchez
Copy link
Collaborator Author

ebruchez commented Dec 9, 2020

If we add the wizard view:

  • Wizard:caseIdsForTopLevelSection
  • Wizard:gatherSectionStatusJava
  • Wizard:gatherTopLevelSectionStatusJava
  • Wizard:isWizardSeparateToc
  • Wizard:normalizeWizardMode
  • Wizard:sectionIdFromCaseIdOpt

@ebruchez
Copy link
Collaborator Author

ebruchez commented Jun 5, 2021

This is done for the JavaScript runtime, including XForms and Form Runner, on the offline branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Development

No branches or pull requests

2 participants