Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DROOLS-5351 - Allow to provide relative resource resolver for DMN runtime #2899

Merged
merged 5 commits into from
May 22, 2020

Conversation

mswiderski
Copy link
Contributor

@tarilabs could you please have a look and let me know if this has any side effects? We would like to pass resources not from the class loader so using relativeResolver seems to work.
In general it makes sense to have the resolver checked first before going via the resource which seems to always be there as it is the dmn resource if I got it right.

If this could be the way to go maybe we could expose the relative resolver as well via DMNRuntimeBuilder...

@tarilabs
Copy link
Member

Hi @mswiderski

seems to always be there

not sure if I understood correctly, but the relativeResolver is only used when on the WB. Anyhow the ordering shouldn't matter. In general any order is fine as long as WB continue to work and kie-server deployment.

Wrt DMNRuntimeBuilder I believe won't be needed as I believe a more natural way should become available once the Kogito codegen are refactored to avoid the "assets sylo-ing" effect.

Jenkins execute full downstream build

@mswiderski
Copy link
Contributor Author

@tarilabs ok, if that is WB only then it might not be future proof approach :)

would it be ok to make the org.kie.dmn.core.compiler.DMNCompilerImpl.processPMMLImport(DMNModelImpl, Import, Function<String, Reader>) protected so we could extend it, mainly this part

Resource relativeResource = resolveRelativeResource(rootClassLoader, model, i, i, relativeResolver); but since it is static we can easily override it. to provide ByteArrayResource instead of UrlResource.

The main reason is that I already have the imported asset loaded so there is no need to resolve it again from class loader/url resource.

@tarilabs
Copy link
Member

allow me to double-check if my memory is serving me well @mswiderski about the relativeResolver... in some circumstances it is totally random! :D

@mswiderski
Copy link
Contributor Author

@tarilabs any additional thoughts on this one?

@tarilabs
Copy link
Member

didnt' have time to go into this sorry.
In the meantime let me understand if it's breaking or just CI

Jenkins execute full downstream build

@tarilabs
Copy link
Member

Jenkins test again

@tarilabs
Copy link
Member

fdb failures unrelated,

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Business Central - Distributions 7.38.0-SNAPSHOT ... SUCCESS [  3.531 s]
[INFO] Business Central Parent ............................ SUCCESS [  0.176 s]
[INFO] Business Central - Webapp Common ................... SUCCESS [  0.950 s]
[INFO] Business Central Deployment Validation ............. SUCCESS [  7.916 s]
[INFO] KIE Theme .......................................... SUCCESS [  0.121 s]
[INFO] KIE Theme - Community .............................. SUCCESS [  3.848 s]
[INFO] KIE Theme - Product ................................ SUCCESS [  3.768 s]
[INFO] Business Central - Home Page - Community ........... SUCCESS [ 24.742 s]
[INFO] Business Central - Monitoring Webapp ............... SUCCESS [13:50 min]
[INFO] Business Central - Webapp .......................... SUCCESS [26:39 min]
[INFO] Business Central - Distribution Wars Parent ........ SUCCESS [  1.724 s]
[INFO] Business Central - Monitoring Distribution Wars .... SUCCESS [01:39 min]
[INFO] Business Central - Distribution Wars ............... SUCCESS [02:01 min]
[INFO] jBPM server distribution ........................... SUCCESS [ 15.453 s]
[INFO] business central add-ons distribution .............. SUCCESS [ 27.426 s]
[INFO] Business Central Tests :: Parent ................... SUCCESS [  0.798 s]
[INFO] Business Central Tests :: REST API ................. SUCCESS [04:14 min]
[INFO] KIE (Drools) Workbench Tests :: GUI 7.38.0-SNAPSHOT  SUCCESS [11:35 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:01 h
[INFO] Finished at: 2020-05-18T21:04:26-04:00
[INFO] ------------------------------------------------------------------------
[WARNING] The requested profile "no-showcase" could not be activated because it does not exist.
+ test 0 -eq 0
[Pipeline] }
Deleting 1 temporary files
[Pipeline] // configFileProvider
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
Deleting 1 temporary files
[Pipeline] // configFileProvider
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // load
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] junit
Recording test results
[Pipeline] archiveArtifacts
Archiving artifacts
Error when executing always post condition:
java.io.IOException: Remote call on kie-psi-rhel7-xlarge-xmem-2771 failed
	at hudson.remoting.Channel.call(Channel.java:961)
	at hudson.FilePath.act(FilePath.java:1072)
	at hudson.FilePath.act(FilePath.java:1061)
	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:233)
	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300)
	at java.lang.StringCoding.encode(StringCoding.java:344)
	at java.lang.String.getBytes(String.java:918)
	at java.io.UnixFileSystem.list(Native Method)
	at java.io.File.list(File.java:1122)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1252)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1206)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1168)
	at org.apache.tools.ant.DirectoryScanner.checkIncludePatterns(DirectoryScanner.java:953)
	at org.apache.tools.ant.DirectoryScanner.scan(DirectoryScanner.java:909)
	at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:500)
	at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:461)
	at hudson.tasks.ArtifactArchiver$ListFiles.invoke(ArtifactArchiver.java:288)
	at hudson.tasks.ArtifactArchiver$ListFiles.invoke(ArtifactArchiver.java:268)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)

[Pipeline] script
[Pipeline] {
[Pipeline] emailext
Sending email to: egonzale@redhat.com noreply@github.com gds.geoffrey.de.smet@gmail.com
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Adding one-line test results to commit status...
Setting status of 92ca7d635cd963f01212fad8d6522568f28010c1 to FAILURE with url https://rhba-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/job/KIE/job/master/job/upstream-downstream/job/drools.downstream/18/ and message: 'Build finished. 64014 tests run, 690 skipped, 0 failed.'
Using context: Linux - full downstream (new)
java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:300)
	at java.lang.StringCoding.encode(StringCoding.java:344)
	at java.lang.String.getBytes(String.java:918)
	at java.io.UnixFileSystem.list(Native Method)
	at java.io.File.list(File.java:1122)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1252)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1282)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1206)
	at org.apache.tools.ant.DirectoryScanner.scandir(DirectoryScanner.java:1168)
	at org.apache.tools.ant.DirectoryScanner.checkIncludePatterns(DirectoryScanner.java:953)
	at org.apache.tools.ant.DirectoryScanner.scan(DirectoryScanner.java:909)
	at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:500)
	at org.apache.tools.ant.types.AbstractFileSet.getDirectoryScanner(AbstractFileSet.java:461)
	at hudson.tasks.ArtifactArchiver$ListFiles.invoke(ArtifactArchiver.java:288)
	at hudson.tasks.ArtifactArchiver$ListFiles.invoke(ArtifactArchiver.java:268)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
Caused: java.io.IOException: Remote call on kie-psi-rhel7-xlarge-xmem-2771 failed
	at hudson.remoting.Channel.call(Channel.java:961)
	at hudson.FilePath.act(FilePath.java:1072)
	at hudson.FilePath.act(FilePath.java:1061)
	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:233)
	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:80)
	at org.jenkinsci.plugins.workflow.steps.CoreStep$Execution.run(CoreStep.java:67)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Finished: FAILURE

but waiting the relaunch of the standard build

@tarilabs
Copy link
Member

@mswiderski I did some checks, and I believe this is fine as long as build becomes green (normal build, failures in fbd to me is unrelated).

This is indeed used mainly by the WB when need to validate a DMN model importing a PMML one, see test: https://github.com/kiegroup/drools/blob/cf139d465922e998fe9322ed9d3e3d93c45cb5ee/kie-dmn/kie-dmn-validation/src/test/java/org/kie/dmn/validation/ValidatorImportTest.java#L214-L216

yet I believe your modification make sense overall, if the resolver is present that shall be given the priority.

@mswiderski
Copy link
Contributor Author

thanks @tarilabs for feedback.

and what about exposing option to provide the resolver via DMNRuntimeBuilder? would this be possible...

I guess a new jira for this is required and then will update commit message and move this to normal PR instead of draft. Just waiting on your comment about DMNRuntimeBuilder

@tarilabs
Copy link
Member

and what about exposing option to provide the resolver via DMNRuntimeBuilder? would this be possible...

I believe I already answered about that in this https://github.com/kiegroup/drools/pull/2899#issuecomment-628668075 ?

If that is "kinda pressing" we can include analogous facility to the DMNRuntimeBuilder as well, but:

  • it will be more in the shape of the test linked above, since models in 2 directories might reference the same relative URI so the resolver function must relate to a given model supplying the relative resource --indeed as similarly to the tests
  • that would work only if using DMNCompilerImpl and not custom compilers

@mswiderski
Copy link
Contributor Author

@tarilabs the thing is that I need to be able to reference some resources from outside and thus having an option to provide this to not be expected on class path is what I am after.

I can make a small prototype for the builder to illustrate what I have in mind, wdyt?

@tarilabs
Copy link
Member

I can make a small prototype for the builder to illustrate what I have in mind, wdyt?

Absolutely. Please keep in mind however that DMNRuntimeBuilder like the javadoc marked since the beginning is an internal utility mainly for Kogito, and while I believe is realistically stable today, there might need changes in the future if Kogito requires.

My original idea is to supply DMNRuntimeBuilder a function in the shape of

(modelNamespace, modelName, relativeImportURI) -> Reader

or analogous.

If you have a prototype to look at we can discuss!

@mswiderski
Copy link
Contributor Author

@tarilabs here you can find the changes that allow to used the resolver via the builder kiegroup@65ff882

@tarilabs
Copy link
Member

@mswiderski thank you for the commit.
Overall the solution is okay based on the premises I mentioned above, but the concern I have is that using simply (relativeURI) -> Reader imho is not enough.

As I tried to explain before in this https://github.com/kiegroup/drools/pull/2899#issuecomment-630698748 I believe the function for resolution shall be (modelNamespace, modelName, relativeImportURI) -> Reader

Let's make an example a "project" like the following:

/myproject
  /taxes
    model1.dmn
    ML.pmml
  /refunds
    model2.dmn
    ML.pmml

in this case the minimal guarantee (for Drools DMN) is that DMN files must have unique combination of namespace+name, but there is no "requirement" for the relative URI to be unique, that for each model could be like "./ML.pmml" but semantically they want to refer to different models.

Would work from your side if we make the function:

from: (relativeURI) -> Reader
  to: (modelNamespace, modelName, relativeImportURI) -> Reader

I can take care of this, once we agree it overall address your needs on your end (I believe it does, since you could always ignore the additional parameters of the function)

@mswiderski
Copy link
Contributor Author

@tarilabs fine with me, I made this to be aligned with the compile method of DMNCompilerImpl to avoid extra changes that I might not be aware of side effects.

Though, what you presented makes sense to me and if that will make it more future proof then go ahead! thanks

@tarilabs
Copy link
Member

@mswiderski let me know if mswiderski#1 works from your side

@mswiderski
Copy link
Contributor Author

@tarilabs sure thing, on it directly, thanks

@mswiderski
Copy link
Contributor Author

@tarilabs works like a charm, shall I merge your PR and update this one?

@tarilabs
Copy link
Member

yes if you merge the mswiderski#1 it shall automatically reflect here

@tarilabs
Copy link
Member

(if you can, remember to merge-with-squash)

@mswiderski
Copy link
Contributor Author

will merge locally and squash so all work is in single commit ... and obviously create jira :)

thanks @tarilabs

@tarilabs
Copy link
Member

to clarify: the reason I would opt for the merge-with-squash in the linked PR, it would just "append" a single commit here. To fully clarify, I'm personally not a fan of the original author always squashing the PR, as this causes "force push" that for long PR under review make reviewers loose track of it :) my2c

@mswiderski mswiderski marked this pull request as ready for review May 20, 2020 09:31
@mswiderski mswiderski changed the title changed order to allow relativeResolver to be invoked first if given DROOLS-5351 - Allow to provide relative resource resolver for DMN runtime May 20, 2020
@mswiderski
Copy link
Contributor Author

@tarilabs as you wish, PR merged into this one and jira created, title updated. All good?

@tarilabs
Copy link
Member

allow me today/tmr to append one simple test, the implementation looks fine for me

@tarilabs
Copy link
Member

@mswiderski please merge-with-squash this mswiderski#2 so to have test coverage and we should be good

@mswiderski
Copy link
Contributor Author

@tarilabs done, thanks!

@tarilabs tarilabs requested a review from hellowdan May 21, 2020 11:05
@tarilabs
Copy link
Member

@mswiderski please also this: mswiderski#3

@mswiderski
Copy link
Contributor Author

done

@sonarcloud
Copy link

sonarcloud bot commented May 21, 2020

Kudos, SonarCloud Quality Gate passed!

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities (and Security Hotspot 0 Security Hotspots to review)
Code Smell A 0 Code Smells

88.0% 88.0% Coverage
0.0% 0.0% Duplication

@mariofusco mariofusco merged commit cee0d36 into apache:master May 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants