integrate mlpm for package deployment

[joemfb]

mlpm is a package manager for XQuery library modules, and REST API extensions/transformations. I've created a (crude) deployment mechanism customizing Roxy's app_specific.rb.

I'd like to discuss some of the tradeoffs in deployment, and an approach for integrating package deployment into Roxy. I'll add more details as soon as I can.

[grtjn]

This should be integrated into deploy modules. Crawl mlpm_modules (configurable path), prepend all subdirs as basedirs, and do usual stuff.

Mlpm_modules should go first. Is order within mlpm important?

Using module locations nice as option. Maybe auto generate into ml-config if placeholder is uncommented.

We could append these mod locations to qconsole app server with special option?

Note group support does not yet include mod location. Only supported on server level now..

[joemfb]

This functionality would be a superset of the proposal in #133.

[dmcassel]

+1

[grtjn]

Would be interesting if the packages would not only contain src and rest-api, but ml-config fragments as well. I prepared setup.xqy to take multiple configs. Then you could really modularize your project..

[joemfb]

thanks @grtjn, I opened joemfb/mlpm#1 to track the possibility of config fragments in packages.

[joemfb]

re: your earlier comments, order is import for REST extensions/transforms, which are parsed and validated when deployed. To be safe, the deployer needs a reverse topological sort of modules and their dependencies.

As for locations, they're supported at both appserver and group levels:

• http://docs.marklogic.com/admin:appserver-add-module-location
• http://docs.marklogic.com/admin:group-add-module-location

[joemfb]

Finally, here's are the deployment notes that I originally promised... I apologize for both the delay, and the length, but there's a lot to consider here. I'd appreciate any feedback y'all can offer ;)

---

mlpm should establish best practices for module authors, and should provide consistency to module consumers. It should also offer flexibility to module authors, requiring as little change as possible to their existing practices.

import mechanisms

There are several ways to import modules in MarkLogic:

• XQuery main <- XQuery lib
  • namespace + absolute path
  • namespace + relative path
  • namespace only (namespace -> absolute path mapped in config)
• JS main <- (XQuery or JS) lib
  • absolute path
  • relative path

Note: there's no way to import JS libs into XQuery main modules.

deployment strategies

There are several potential strategies for module deployment:

• flatten dependencies, identify by name only

The simplest strategy, provides a consistent absolute path. Easily compatible with namespace-only imports. Easily updated via semver. Modules with dependencies should import via absolute path.

Downside: disallows concurrent use of multiple module versions; requires a bower-style conflict resolution if different module versions are required

• flatten dependencies, identify by name/version

Allows concurrent use of multiple modules versions.

Downside: doesn't provide a consistent path; not easily updated via semver; namespace URIs must include version info to allow for namespace-only imports

• nest dependencies

Allows concurrent use of multiple modules versions. Easily updated via semver.

Downside: incompatible with namespace-only imports; all imports must be via relative path.

• magic

Rewrite import paths when modules are installed. Upside: support any strategy.

Downside: magic

conclusion

Given these options, I favor the first; I think it's the most easily understood. Rewriting import paths has the least impact on authors, but would have substantial and difficult-to-debug failure modes for consumers. As for multiple concurrent module versions: I haven't come seen it in any XQuery. At the same time, I would guess that's partly a function of the lack of XQuery package management. I do think it would be useful, but I don't think it's essential. The cost of supporting it seems high, given how XQuery module imports work.

As for SJS, I favor absolute over relative paths. If, in the future, require is modified to support recursive module lookups, than nested dependencies will become natural.

Thoughts?

[grtjn]

Quick note on mod locations: yes supported by admin lib, but not yet in ml-config..

More later.. 

[rlouapre]

@grtjn can you clarify what's not supported yet?

Module location at application server level is supported by Roxy.

[joemfb]

This confusion is my fault: I confused @grtjn's comment re: the Roxy bootstrap module location support with MarkLogic's general capability.

The gist I linked earlier (the "crude" deployment mechanism for modules: https://gist.github.com/joemfb/c786696f459290e57c73) automatically creates appserver-level module location bindings, bypassing the Roxy mechanism.

[grtjn]

@rlouapre see first comment above: module locations at group level, still on my todo list.. ;)

[grtjn]

@joemfb haven't read your entire comment yet, but wouldn't it simplify the deployment if we would deploy all src modules first, also of dependencies, and deploy all rest extensions last? Modules are not checked at upload, extensions are but extensions cannot depend on each other (not through import at least)..

Might just work, and saves a lot of headache around dependency checks.. :P

[joemfb]

That's the approach taken by the linked gist, and it should be sufficient for most use-cases. However REST extensions/transformations are library modules, and could be imported by other modules (although it's probably not recommended). Finally, there are REST assets (http://docs.marklogic.com/guide/rest-dev/extensions#id_55309), which I haven't really examined in any depth yet ...

A topo-sort isn't completely necessary, but would be more "correct", and also fun ;)

[grtjn]

Ah right, assets, that is new. I think it is another way of uploading modules. Do src first, assets second, extensions/transforms last. That would be a good first step.

I would normaly highly discourage importing REST extensions, but I may have been guilty on doing similar things with MLCP transforms, decorators, custom facets, etc.. ;-)

[grtjn]

About time to add this to Roxy..

[grtjn]

Simple first step would be to support ./ml local deploy mlpm. That would basically wrap around mlpm deploy to add the -u -p -H -P parameters, like we do for mlcp..

[joemfb]

see the README for an app_specific.rb snippet that does just that: https://github.com/joemfb/mlpm. However, that kind of run-time dependency on the mlpm client might not be preferable long term ...

[grtjn]

Pretty close indeed. Needs prompt_user/pass, and a slightly tighter integration into deploy modules perhaps.

Doing a sys-call is questionable, but we do the same for mlcp, xqsync, recordloader, etc. Not expecting changes currently either, are we?

[joemfb]

Shouldn't need to prompt: the credentials used for module deployment are sufficient. Not sure about tighter integration, I don't think you want to deploy packages every time you deploy modules. Open to debate I guess.

I'd like to see it be really easy to use: ie, use the mlpm client if you need to manage dependencies, but don't have to have it just to deploy an app with dependencies. The client also exposes an API, so I guess roxy could bundle a node script (and run npm install) instead of requiring the executable.

[grtjn]

Prompt_password is an internal function that prompts if roxy password prop is left blank. ;-)

I'd include it in deploy modules for symmetry with clean modules. The deploy mod is not very smart so far. You could deploy src+rest to skip it. And make it look for mlpm_modules dir for starters. If that dir exists mlpm must be installed. :-)

[grtjn]

Need to reread above comments, but wanted to mention I have code for a deploy_packages that leverages the internal deploy commands to mimic mlpm deploy. Not perfect, but close enough to get you going..

[grtjn]

I'll open a PR for that..

[joemfb]

See also https://gist.github.com/joemfb/c786696f459290e57c73. It's somewhat out of date, but could be helpful.

[grtjn]

Interesting, totally glanced over that! I had this so far:

  def deploy_packages
    ### preserve org settings
    org_src_dir = @properties['ml.xquery.dir']
    org_mod_prefix = @properties['ml.modules-prefix']
    org_ext_dir = @properties['ml.rest-ext.dir']
    org_options_dir = @properties['ml.rest-options.dir']
    org_transforms_dir = @properties['ml.rest-transforms.dir']

    ### upload everything blindly as module first
    @properties['ml.xquery.dir'] = File.expand_path("mlpm_modules")
    @properties['ml.modules-prefix'] = "/ext/mlpm_modules"
    # leverage existing deploy functionality
    deploy_src

    ### go over second time to find REST transforms, and REST extensions
    @properties['ml.rest-ext.dir'] = File.expand_path(".")
    @properties['ml.rest-options.dir'] = File.expand_path(".")
    @properties['ml.rest-transforms.dir'] = File.expand_path(".")
    # leverage existing deploy functionality
    Dir.glob("mlpm_modules/**/*") do |file|
      if File.file?(file)
        content = File.read file
        ARGV << "--file=#{file}"
        # recognize REST transforms by module namespace, or by xsl:param
        if content.include? 'http://marklogic.com/rest-api/transform/' or content.include? '<xsl:param name="context" as="map:map"/>'
          deploy_transform
        # recognize REST extensions by module namespace
        elsif content.include? 'http://marklogic.com/rest-api/resource/'
          deploy_ext
        end
        ARGV.pop
      end
    end

    ### restore org settings
    @properties['ml.xquery.dir'] = org_src_dir
    @properties['ml.modules-prefix'] = org_mod_prefix
    @properties['ml.rest-ext.dir'] = org_ext_dir
    @properties['ml.rest-options.dir'] = org_options_dir
    @properties['ml.rest-transforms.dir'] = org_transforms_dir
  end

[grtjn]

Your code is probably more complete/accurate.. :P

[joemfb]

My concern is duplicated implementation/effort, and creeping incompatibility. I've thought about having mlpm maintain some kind of metadata file laying out the order, structure, and type of assets within packages. Then it'd be easy for various tools to deploy; they'd just be following that file instead of parsing, classifying, sorting assets and packages.

[grtjn]

Hmm, something like adding extra details to mlpm.json on publish? That could be useful. You would need to touch existing versions perhaps, but that is doable too..

Duplication, yeah totally. I completely missed that link to your work, even though you mentioned in the very first comment.. :p

Doing a sys-call prevents incompatibilities, and by far easiest, but not really fond of it..

[grtjn]

Sounds like this needs more discussion, pushing away..

[grtjn]

After pondering a while about this, I ended up doing this in slush-marklogic-node: https://github.com/marklogic/slush-marklogic-node/blob/master/app/templates/deploy/app_specific.rb#L77

I defer the complexity of deploying MLPM package to mlpm itself, which because of that needs to be installed. Then again, you need to have it anyhow to pull down the packages anyhow.

I'm also doing deploy_packages before deploy_modules, since src and rest extensions of the project can depend on MLPM packages, but not other way around.

MLPM packages are still deployed in alphabetic or random order probably, but I haven't seen packages with complex dependencies yet.

I think it is time to integrate the slush approach into Roxy itself..

[joemfb]

FYI, mlpm deploys packages in inverse topological order, based on any dependencies between them: https://github.com/joemfb/mlpm/blob/master/lib/project.js#L151

[grtjn]

Awesome, even better!