Duplicate entity definition when deploying with gradle

[burleson]

The issue

Currently on DHF 4.0.1

When deploying using either mlLoadModules or mlDeploy we are running into an issue where our entity definition xml is being deployed twice. When we attempt to run our ingestion it appears that a part of the DHF framework is pulling both of these definitions simultaneously and results in a sequence error.

Tech details

We're currently on ML 9.0-7 and DHF 4.0.1

The devil is in the details...

Example location of our entity xml is ./plugins/entities/MDM/input/MDMInput.xml

When we run mlLoadModules or mlDeploy this results in the following two documents to be inserted into the modules database:

/entities/MDM/input/MDMImport/MDMImport.xml
/MDM/input/MDMImport/MDMImport.xml

When attempting to run an ingestion- we get the following error:

2018-11-13 05:00:08.624 Notice: XDMP-AS: (err:XPTY0004) flow:get-flow-nocache($entity-name, $flow-name, $flow-type) -- Invalid coercion: ( MDMImport..., &#10$
2018-11-13 05:00:08.624 Notice:+in /data-hub/4/impl/flow-lib.xqy, at 98:31,
2018-11-13 05:00:08.624 Notice:+in flow:get-flow-nocache("MDM", "MDMImport", "input") [1.0-ml]
2018-11-13 05:00:08.624 Notice:+ $entity-name = "MDM"
2018-11-13 05:00:08.624 Notice:+ $flow-name = "MDMImport"
2018-11-13 05:00:08.624 Notice:+ $flow-type = "input"
2018-11-13 05:00:08.624 Notice:+in /data-hub/4/impl/flow-lib.xqy, at 90:8,
2018-11-13 05:00:08.624 Notice:+in flow:get-flow("MDM", "MDMImport", "input") [1.0-ml]
2018-11-13 05:00:08.624 Notice:+ $entity-name = "MDM"
2018-11-13 05:00:08.624 Notice:+ $flow-name = "MDMImport"
2018-11-13 05:00:08.624 Notice:+ $flow-type = "input"
2018-11-13 05:00:08.624 Notice:+ $duration = xs:dayTimeDuration("PT10S")
2018-11-13 05:00:08.624 Notice:+ $key = "flow-cache-MDMMDMImportinput"
2018-11-13 05:00:08.624 Notice:+ $flow = ()
2018-11-13 05:00:08.624 Notice:+in /data-hub/4/transforms/run-flow.xqy, at 53:17,
2018-11-13 05:00:08.624 Notice:+in function() as item()() [1.0-ml]
2018-11-13 05:00:08.624 Notice:+ $params = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)
2018-11-13 05:00:08.624 Notice:+ $job-id = "mdm-import-30a1f8be-1d59-4a77-8fac-970f03ca53eb"
2018-11-13 05:00:08.624 Notice:+ $entity-name = "MDM"
2018-11-13 05:00:08.624 Notice:+ $flow-name = "MDMImport"
2018-11-13 05:00:08.624 Notice:+ $uri = "/P6-XXSUF_PROJECT_INFO/Project/178468.xml"
2018-11-13 05:00:08.624 Notice:+in /data-hub/4/impl/perflog-lib.xqy, at 34:9,
2018-11-13 05:00:08.624 Notice:+in perf:log("/transforms/run-flow:transform", function() as item()) [1.0-ml]
2018-11-13 05:00:08.624 Notice:+ $what = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)
2018-11-13 05:00:08.624 Notice:+ $func = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)
2018-11-13 05:00:08.624 Notice:+in /data-hub/4/transforms/run-flow.xqy, at 48:2,
2018-11-13 05:00:08.624 Notice:+in xdmp:function(fn:QName("http://marklogic.com/rest-api/transform/run-flow","transform"), "/data-hub/4/transforms/run-flow.xqy")(map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-inst$
2018-11-13 05:00:08.624 Notice:+ $context = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)
2018-11-13 05:00:08.624 Notice:+ $params = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)
2018-11-13 05:00:08.624 Notice:+ $content = document{<P6_ID>178468</P6_ID>...}
2018-11-13 05:00:08.624 Notice:+in /data-hub/4/rest-api/lib/extensions-util.xqy, at 943:44,
2018-11-13 05:00:08.624 Notice:+in extut:call-service("ml:inputFlow", "TRANSFORM", xdmp:function(fn:QName("http://marklogic.com/rest-api/transform/run-flow","transform"), "/data-hub/4/transforms/run-flow.xqy"), map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema"$
2018-11-13 05:00:08.624 Notice:+ $extension-name = "ml:inputFlow"
2018-11-13 05:00:08.624 Notice:+ $method = "TRANSFORM"
2018-11-13 05:00:08.624 Notice:+ $service = xdmp:function(fn:QName("http://marklogic.com/rest-api/transform/run-flow","transform"), "/data-hub/4/transforms/run-flow.xqy")
2018-11-13 05:00:08.624 Notice:+ $context = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)
2018-11-13 05:00:08.624 Notice:+ $service-params = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)
2018-11-13 05:00:08.624 Notice:+ $input = document{<P6_ID>178468</P6_ID>...}
2018-11-13 05:00:08.624 Notice:+in /data-hub/4/rest-api/lib/extensions-util.xqy, at 899:20,
2018-11-13 05:00:08.624 Notice:+in function() as item()() [1.0-ml]
2018-11-13 05:00:08.624 Notice:+in /data-hub/4/rest-api/lib/extensions-util.xqy,
2018-11-13 05:00:08.624 Notice:+in xdmp:invoke(function() as item(), different-transaction...) [1.0-ml]
2018-11-13 05:00:08.624 Notice:+in /data-hub/4/rest-api/lib/extensions-util.xqy, at 897:12,
2018-11-13 05:00:08.624 Notice:+in extut:invoke-service("ml:inputFlow", "TRANSFORM", (), xdmp:function(fn:QName("http://marklogic.com/rest-api/transform/run-flow","transform"), "/data-hub/4/transforms/run-flow.xqy"), map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLS$
2018-11-13 05:00:08.624 Notice:+ $extension-name = "ml:inputFlow"
2018-11-13 05:00:08.624 Notice:+ $method = "TRANSFORM"
2018-11-13 05:00:08.624 Notice:+ $default-txn-mode = ()
2018-11-13 05:00:08.624 Notice:+ $service = xdmp:function(fn:QName("http://marklogic.com/rest-api/transform/run-flow","transform"), "/data-hub/4/transforms/run-flow.xqy")
2018-11-13 05:00:08.624 Notice:+ $context = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)
2018-11-13 05:00:08.624 Notice:+ $service-params = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)
2018-11-13 05:00:08.624 Notice:+ $input = document{<P6_ID>178468</P6_ID>...}
2018-11-13 05:00:08.624 Notice:+ $in-txn = fn:false()
2018-11-13 05:00:08.624 Notice:+ $txn-curr = "update"
2018-11-13 05:00:08.624 Notice:+ $txn-mode = "query"
2018-11-13 05:00:08.624 Notice:+in /data-hub/4/rest-api/models/transform-model.xqy, at 315:13,
2018-11-13 05:00:08.624 Notice:+in tformod:apply-transform("ml:inputFlow", map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>), map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="h$
2018-11-13 05:00:08.624 Notice:+ $transform-name = "ml:inputFlow"
2018-11-13 05:00:08.624 Notice:+ $context = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)
2018-11-13 05:00:08.624 Notice:+ $params = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)
2018-11-13 05:00:08.624 Notice:+ $content = document{<P6_ID>178468</P6_ID>...}
2018-11-13 05:00:08.624 Notice:+in /data-hub/4/rest-api/models/document-model-update.xqy, at 3895:29,
2018-11-13 05:00:08.624 Notice:+in docmodupd:write-bulk-documents(xdmp:get-request-part-headers#0, xdmp:get-request-part-body#1, "ml:inputFlow", map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>), (),$
2018-11-13 05:00:08.624 Notice:+ $header-getter = xdmp:get-request-part-headers#0
2018-11-13 05:00:08.624 Notice:+ $body-getter = xdmp:get-request-part-body#1
2018-11-13 05:00:08.624 Notice:+ $trans-name = "ml:inputFlow"
2018-11-13 05:00:08.624 Notice:+ $trans-params = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)
2018-11-13 05:00:08.624 Notice:+ $forest-id = ()
2018-11-13 05:00:08.624 Notice:+ $default-metadata = map:map()
2018-11-13 05:00:08.624 Notice:+ $document-metadata = map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" .../>)

[srinathgit]

1. The entity file has to be a json file , in this case it should be "MDM.entity.json" and this file should be present in plugins/entities/MDM directory.
2. The /plugins/entities/MDM/input/ directory would have the plugins (main, content, headers, triples and properties file describing the flow)
3. https://marklogic.github.io/marklogic-data-hub/understanding/project-structure/ has more information on the project structure

Can you please try this and let me know if it works ?

[burleson]

We have the MDM.entity.json file. I've attached the directory for this input flow so you can take a better look

MDM.zip

[srinathgit]

Based on the flow you had provided, the project seems to be 1.x version and not 4.0.1. You can do the following:

1. Create a new input flow in 4.0.1. If you are not using entity services, you will have to explicitly set –Pusees=false .
2. Reconcile your current plugins with 4.0.1 plugins (return type etc may be different between versions).
3. Then you can deploy and run the flows

[srinathgit]

Mike responded in an email saying that the provided suggestion helped resolve the issue. Closing the case.