Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate FIX into Metafacture CLI / Use FIX as additional library #90

Closed
TobiasNx opened this issue Dec 7, 2021 · 16 comments
Closed

Comments

@TobiasNx
Copy link
Collaborator

TobiasNx commented Dec 7, 2021

At the moment Metafacture Fix is only available as library for Metafacture in Java.

I would suggest that we also find a solution to integrate it as library into Metafacture CLI.

If you could install FIX as additional library to the command line programm then this would allow users of metafacture CLI to use it before metafacture-fix is part of metafacture-core. It would allow Metafacture CLI users to test FIX.

@fsteeg says that Metafacture does not support adding new libraries to the CLI-programm in general.

Catmandu has the option to install additional packages/libraries to add new featrues to the default features of Catmandu. https://librecat.org/Catmandu/#installation

For me personally it would make it easier to review new features in Metafacture-Fix since I am not depending on a full java project to test things. I also could create a repo with scenarios that are not working that could be tried. Then I could review as I do in metafacture-core tickets.

This would need an additional feature to allow install of libraries from branches similar to: https://github.com/metafacture/metafacture-core#building-metafacture-core-from-source

TL;DR
Support installing FIX as library to metafacture-core CLI.
Allpw installing from different branches.

@TobiasNx TobiasNx changed the title Integrate FIX into Metafacture CLI Integrate FIX into Metafacture CLI / Use FIX as additional library Dec 7, 2021
@blackwinter
Copy link
Member

@fsteeg says that Metafacture does not support adding new libraries to the CLI-programm in general.

But it does? You can place the (locally built) metafacture-fix jar in the plugins/ directory of your metafacture-runner distribution.

@fsteeg
Copy link
Member

fsteeg commented Dec 7, 2021

You can place the (locally built) metafacture-fix jar in the plugins/ directory

Oh right, I completely forgot about that. So that would be ./gradlew shadowJar → creates a jar in metafix/build/libs/. So e.g. being on the master branch in metafacture-fix, this should update a local metafacture distribution at ~/metafacture/: ./gradlew shadowJar ; cp metafix/build/libs/metafix-0.2.0-SNAPSHOT.jar ~/metafacture/plugins.

@TobiasNx
Copy link
Collaborator Author

TobiasNx commented Dec 7, 2021

Cool, Thanks @blackwinter So this works on a branch in metafacture-fix too?

@blackwinter
Copy link
Member

Yes, you can build the jar from any branch: just check out the branch and build (see @fsteeg's comment).

@TobiasNx
Copy link
Collaborator Author

TobiasNx commented Dec 7, 2021

Thanks, I think the possibility to integrate plugins should be documented in the wiki of metafacture-core and also here in the metafacture-fix ReadMe.

I updated the "installation"-command so that swichting between version and updating to other branch brings no conflicts:

First installation:
$ ./gradlew shadowJar ; cp metafix/build/libs/metafix-*-SNAPSHOT.jar ~/metafacture-core/plugins

Following installations:
$ rm ~/metafacture-core/plugins/metafix-*-SNAPSHOT.jar ; ./gradlew shadowJar ; cp metafix/build/libs/metafix-*-SNAPSHOT.jar ~/metafacture-core/plugins

@fsteeg fsteeg removed their assignment Dec 7, 2021
@TobiasNx
Copy link
Collaborator Author

TobiasNx commented Dec 8, 2021

okay, I tried to set it up. using a CLI version from a source build runner: metafacture-core/metafacture-runner/build/install/metafacture-core/plugins (fluxDev is my alias for flux.sh)

i get following error when running a flux:

$ fluxDev 'test1_externalMapInline.flux'
Exception in thread "main" org.metafacture.commons.reflection.ReflectionException: class could not be instantiated: class org.metafacture.metafix.Metafix
        at org.metafacture.commons.reflection.ConfigurableClass.newInstance(ConfigurableClass.java:140)
        at org.metafacture.commons.reflection.ObjectFactory.newInstance(ObjectFactory.java:105)
        at org.metafacture.flux.parser.FluxProgramm.createElement(FluxProgramm.java:78)
        at org.metafacture.flux.parser.FluxProgramm.addElement(FluxProgramm.java:90)
        at org.metafacture.flux.parser.FlowBuilder.pipe(FlowBuilder.java:736)
        at org.metafacture.flux.parser.FlowBuilder.flowtail(FlowBuilder.java:514)
        at org.metafacture.flux.parser.FlowBuilder.flow(FlowBuilder.java:226)
        at org.metafacture.flux.parser.FlowBuilder.flux(FlowBuilder.java:122)
        at org.metafacture.flux.FluxCompiler.compileFlow(FluxCompiler.java:66)
        at org.metafacture.flux.FluxCompiler.compile(FluxCompiler.java:54)
        at org.metafacture.runner.Flux.main(Flux.java:85)
Caused by: java.lang.reflect.InvocationTargetException
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
        at org.metafacture.commons.reflection.ConfigurableClass.newInstance(ConfigurableClass.java:135)
        ... 10 more
Caused by: java.lang.NoClassDefFoundError: org/eclipse/xtext/ISetup
        at java.base/java.lang.ClassLoader.defineClass1(Native Method)
        at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
        at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
        at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:550)
        at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:458)
        at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:452)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:451)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        at java.base/java.lang.ClassLoader.defineClass1(Native Method)
        at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
        at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
        at java.base/java.net.URLClassLoader.defineClass(URLClassLoader.java:550)
        at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:458)
        at java.base/java.net.URLClassLoader$1.run(URLClassLoader.java:452)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:451)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        at org.metafacture.metafix.Metafix.buildPipeline(Metafix.java:118)
        at org.metafacture.metafix.Metafix.<init>(Metafix.java:99)
        at org.metafacture.metafix.Metafix.<init>(Metafix.java:91)
        at org.metafacture.metafix.Metafix.<init>(Metafix.java:87)
        ... 15 more
Caused by: java.lang.ClassNotFoundException: org.eclipse.xtext.ISetup
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        ... 39 more

Any idea why this is not working?

@fsteeg
Copy link
Member

fsteeg commented Dec 8, 2021

Unassigning since we never made this ready. If it's not actually working, I suggest you use a different way of testing for now @TobiasNx and we discuss when to address this in the future.

@TobiasNx
Copy link
Collaborator Author

TobiasNx commented Dec 8, 2021

Okay agree. I will use another way and wait for it being ready. (I assigned since it was said that there is a easy way to integrate plugins.)

When making it ready we should keep in mind if this a general problem of metafacture or a problem of the fix module.

@blackwinter
Copy link
Member

blackwinter commented Dec 8, 2021

The task :metafix:jar doesn't create a fat jar, so it doesn't include the dependencies.

We can add a new task for that:

diff --git metafix/build.gradle metafix/build.gradle
index db7e978..8f884f2 100644
--- metafix/build.gradle
+++ metafix/build.gradle
@@ -1,5 +1,6 @@
 plugins {
   id 'com.adarshr.test-logger' version '1.7.0'
+  id 'com.github.johnrengelman.shadow' version '4.0.1'
   id 'maven-publish'
 }
 
@@ -28,6 +29,8 @@ dependencies {
   testImplementation "org.mockito:mockito-junit-jupiter:${versions.mockito}"
 }
 
+apply plugin: 'com.github.johnrengelman.shadow'
+
 configurations {
   mwe2 {
     extendsFrom implementation

The invocation would then be:

./gradlew -p metafix clean shadowJar

NOTE: Those changes alone are not sufficient long-term, because they interact badly with :metafix:jar and :metafix-ide:shadowJar.

UPDATE: Doesn't work either 😞

java.lang.NoSuchFieldError: EOF_TOKEN
        at org.eclipse.xtext.parser.antlr.Lexer.nextToken(Lexer.java:59)
        at org.antlr.runtime.BufferedTokenStream.fetch(BufferedTokenStream.java:143)
        at org.antlr.runtime.BufferedTokenStream.sync(BufferedTokenStream.java:137)
        at org.antlr.runtime.BufferedTokenStream.fill(BufferedTokenStream.java:286)
        at org.antlr.runtime.BufferedTokenStream.toString(BufferedTokenStream.java:251)
        at org.eclipse.xtext.parser.antlr.AbstractInternalAntlrParser.parse(AbstractInternalAntlrParser.java:553)
        at org.eclipse.xtext.parser.antlr.AbstractAntlrParser.doParse(AbstractAntlrParser.java:102)
        at org.eclipse.xtext.parser.antlr.AbstractAntlrParser.parse(AbstractAntlrParser.java:84)
        at org.eclipse.xtext.parser.antlr.AbstractAntlrParser.doParse(AbstractAntlrParser.java:62)
        at org.eclipse.xtext.parser.AbstractParser.parse(AbstractParser.java:33)
        at org.eclipse.xtext.resource.XtextResource.doLoad(XtextResource.java:176)
        at org.eclipse.xtext.linking.lazy.LazyLinkingResource.doLoad(LazyLinkingResource.java:114)
        at org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(ResourceImpl.java:1518)
        at org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(ResourceImpl.java:1297)
        at org.eclipse.xtext.resource.persistence.StorageAwareResource.load(StorageAwareResource.java:79)
        at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLoad(ResourceSetImpl.java:259)
        at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLoadHelper(ResourceSetImpl.java:274)
        at org.eclipse.xtext.resource.XtextResourceSet.getResource(XtextResourceSet.java:265)
        at org.eclipse.xtext.resource.SynchronizedXtextResourceSet.getResource(SynchronizedXtextResourceSet.java:25)
        at org.metafacture.metafix.validation.XtextValidator.getResource(XtextValidator.java:61)
        at org.metafacture.metafix.validation.XtextValidator.getValidatedResource(XtextValidator.java:65)
        at org.metafacture.metafix.FixStandaloneSetup.parseFix(FixStandaloneSetup.java:40)
        at org.metafacture.metafix.Metafix.buildPipeline(Metafix.java:108)
        at org.metafacture.metafix.Metafix.<init>(Metafix.java:89)
        at org.metafacture.metafix.Metafix.<init>(Metafix.java:81)
        at org.metafacture.metafix.Metafix.<init>(Metafix.java:77)
        ... 15 more

@TobiasNx
Copy link
Collaborator Author

TobiasNx commented May 20, 2022

I think we should make the additional integration of FIX as library for the CLI possible before the Bibliothekskongress.
What do you think @fsteeg and @blackwinter ?

If we introduce FIX to the world it should be usable beyond JAVA and the playground.

@blackwinter
Copy link
Member

I definitely support the effort, but I don't think I'll be able to do it. There are more pressing issues w.r.t. our pipeline that I have to address before Go Live (performance still being one of them). So I'm unassigning myself.

@blackwinter blackwinter removed their assignment May 20, 2022
@fsteeg
Copy link
Member

fsteeg commented May 20, 2022

If we introduce FIX to the world it should be usable beyond JAVA and the playground.

That's why I made the metafix-runner part of the 0.2.0 release, it allows users to run fix workflows on the command line, see https://github.com/metafacture/metafacture-fix/releases/

Making it available in the core distribution remains our longer-term goal (see also 659a551#r74105804).

@dr0i
Copy link
Member

dr0i commented Mar 26, 2024

Is this not solved?
Here is how it works for me, testing some new things not yet released, version in build.gradle is set to 1.0.1-SNAPSHOT:

./gradlew distZip
unzip ./metafix-runner/build/distributions/metafix-runner-1.0.1-SNAPSHOT.zip
# test it on a flux:
metafix-runner-1.0.1-SNAPSHOT/bin/metafix-runner ./metafix/src/test/resources/org/metafacture/metafix/integration/lookup/fromXml/toJson/lookupInDeeplyNestedArrayOfObjects/test.flux
cat ./metafix/src/test/resources/org/metafacture/metafix/integration/lookup/fromXml/toJson/lookupInDeeplyNestedArrayOfObjects/output-metafix.json

@dr0i dr0i assigned TobiasNx and unassigned fsteeg Mar 26, 2024
@blackwinter
Copy link
Member

FYI: You don't need to create the distribution, you can execute the runner directly.

./gradlew :metafix-runner:run --args="$PWD/path/to.flux"

@TobiasNx
Copy link
Collaborator Author

This is a "problem" of fix as a separate repo. I haven't checked if it is still a problem, but as a workaround @fsteeg created a separate runner for fix.

I do not know if it needs to be fixed until fix will be integrated into core.

@TobiasNx
Copy link
Collaborator Author

I see this as a wont fix. The workaround with the fix runner and the CLI version that you described in your PR is okay for me!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

No branches or pull requests

4 participants