Skip to content

How modules are loaded

rjrudin edited this page Apr 23, 2019 · 27 revisions

By default, ml-gradle loads modules from src/main/ml-modules. This directory aligns with the Maven convention of placing all application code under "src/main", and "ml-modules" was chosen as a MarkLogic-specific path.

To accommodate storing both REST API specific modules (such as options, services, and transforms) and "regular" application modules under the same parent directory, the following directory structure is used for identifying where different kinds of modules should be stored:

  1. /root - You store any modules here that aren't REST API options, services, transforms, or namespaces. The module will be loaded with a URI relative to "/root" (it won't have "/root" in the URI).
  2. /ext - You can also store any modules here too, just like "/root". What's the difference? "/ext" was actually supported first in an effort to mirror what the REST API suggests with its /v1/ext endpoint. But in practice, there's little value in storing modules under "/ext", so it's easier just to toss everything under "/root".
  3. /options - REST API search options are stored here. The name of the file (minus the extension) is used to name the search options loaded into ML.
  4. /services - REST API resource extensions are stored here. The name of the file (minus the extension) is used to name the resource extension.
  5. /transforms - REST API transforms are stored here. The name of the file (minus the extension) is used to name the transform.
  6. /namespaces - A REST API namespace namespace can be registered via a file containing the namespace URI, and the name of the file (minus the extension, which can be anything) is used for the namespace prefix.
  7. In addition, any "unrecognized" directory under ml-modules - i.e. a directory that isn't one of the above - will have its contents loaded into the modules database. Prior to ml-gradle 3.0.0, the URI of each such document would include the name of the unrecognized directory. But with version 3.0.0 and greater, the URI does not have the unrecognized directory name. Best practice is to stick with "ext" and "root" for modules and not use this feature!

Modules are loaded when you run "mlDeploy". They can also be loaded by themselves (note that as described below, mlLoadModules only loads modules that have been created or modified since mlLoadModules was last run - mlReloadModules ensures everything is loaded):

gradle mlLoadModules

It's often helpful to use info-level logging so you can see which modules are loaded:

gradle -i mlLoadModules

And debug-level logging will show all the directories that ml-gradle looks in for modules:

gradle -d mlLoadModules

You can also reload modules - i.e. clear the modules database first, and then load modules:

gradle mlReloadModules

If you run into any problems, please see Debugging module loading.

Ports used for loading modules

ml-gradle has to handle REST modules - options, transforms, services, and namespaces - differently from non-REST modules, as REST modules must be loaded via an app-specific REST server. However, not every MarkLogic app has a REST server. For this reason, ml-gradle takes the following approach by default:

  1. Non-REST modules are loaded via the port defined by mlAppServicesPort, which defaults to 8000. This port is nearly guaranteed to exist in every ML cluster, making it a safe choice.
  2. REST modules are loaded via the port defined by mlRestPort, which does not have a default value.

See the Property reference for all of the properties that are used for connecting to these ports.

This is an important distinction for when you run into an error while loading modules, as your non-REST modules may be loading fine, but your REST modules are not.

Replacing tokens in modules

As described in the Configuring resources page, a "custom tokens" map can be populated to specify tokens in resource payloads that will be replaced. These tokens will also be replaced in module files - but as of version 3.2.0, this is only done in "asset" modules, not yet REST API modules (services, options, and transforms).

Two properties control this behavior that are worth noting:

  1. mlReplaceTokensInModules - controls whether tokens are replaced or not, defaults to true
  2. mlUseRoxyTokenPrefix - (see the note below about how this property changes in 3.2.0) expects tokens to start with "@ml.", mirroring Roxy's behavior. This defaults to true; if you don't need these prefixes in place, be sure to set this property to false.

NEW in version 3.2.0 - as described on the Configuring resources page, all Gradle properties will be added to the tokens map by default. And, mlUseRoxyTokenPrefix now defaults to false, effectively deprecating that feature. You can emulate that behavior by setting the following properties to mirror Roxy's behavior:

mlTokenPrefix=@ml.
mlTokenSuffix=

Thus, in 3.2.0, with mlPropsAsTokens and mlReplaceTokensInModules both defaulting to true, the default behavior is that all Gradle properties are used as tokens (with "%%" as the default prefix and suffix), and these tokens will be replaced in modules if they exist.

Only loading modules that need loading

See Watching for module changes for more information on this topic.

On a project with REST extensions, loading all of an application's modules can take many seconds, as each REST extension must be loaded in a separate call to the MarkLogic Client REST API. This isn't too painful for initializing an empty modules database, but it's unacceptable during a code-test cycle.

To address this, ml-gradle keeps track of when it last loaded each module. This is done via a properties file in the "build" directory of the project (so that the properties file is not in version control). The path of the file is build/ml-javaclient-util/module-timestamp.properties. When ml-gradle tries to load a module, it first checks to see if the module's last-modified timestamp is greater than what is recorded in the properties file. If so, the module is loaded, and the properties file is updated. If not, the module is not loaded.

As of 3.3.1, this feature can be disabled by setting the mlModuleTimestampsPath property to an empty string. Prior to 3.3.1, it can be disabled in the following manner in build.gradle:

ext {
  mlAppConfig.setModuleTimestampsPath(null)
}

In practice though, it's common that you'll either run "mlReloadModules" to clear a modules database and load all modules (which ensures that any modules deleted from a project are also deleted from the modules database), or you'll run "mlWatch" to load files as they're created/modified.

Support for multiple environments

Starting in 3.14.0, ml-gradle will include the name of the host that modules are loaded via in the property keys in the module timestamps file. This allows you to e.g. run mlLoadModules against different environments from the same build and still get modules loaded into each modules database. If you wish to disable this behavior, just set the following property in gradle.properties:

mlModuleTimestampsUseHost=false

Copying search options to additional app servers

Search options are different from other REST extensions in that they are only made available via the REST server that was used to load them. So for example, if you have a DHF project, and you have a set of search options you'd like to be available via both the staging and final REST servers - well, there's not an easy way to do that. The options you define under src/main/ml-modules/options will be loaded via the final REST server, and thus not available via the staging REST server.

Version 3.12.0 of ml-gradle introduces a new task for copying search options to address this problem - CopySearchOptionsTask:

task copyMyOptions(type: com.marklogic.gradle.task.client.CopySearchOptionsTask) {
  mustRunAfter mlLoadModules // This ensures that when running mlReloadModules, modules are loaded first
  sourceServer = "data-hub-FINAL"
  targetServer = "data-hub-STAGING"
  optionsFilename = "my-options.json"
  // Can set a "group" property if the servers do not belong to the Default group
  // Can set a "client" property to use a DatabaseClient other than the one returned by mlAppConfig.newModulesDatabaseClient()
}

After defining an instance of CopySearchOptionsTask (can define many as needed), you can attach it to the end of a deployment:

mlPostDeploy.dependsOn copyMyOptions

Unfortunately there's not an easy way to integrate this into mlWatch, nor attach it to mlLoadModules or mlReloadModules. But at least for the latter, you can always run both task:

gradle mlReloadModules copyMyOptions

Or just make a custom task that calls those together.

Defining metadata for resources and transforms

The Client REST API allows for metadata to be defined when loading resources and transforms. See Service metadata in the ml-javaclient-util project for more information.

You can’t perform that action at this time.