Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to handle Neo4j connections with the PDI EE repository #138

Closed
jfmonteil opened this issue Sep 20, 2019 · 8 comments

Comments

@jfmonteil
Copy link

commented Sep 20, 2019

While using Pentaho 8.3 with the database repository, i have the following error

Namespace Neo4j does not exist in the repository
2019-09-19_17h36_13

I had the following comment form Hitachi developpers

What I see happening first is that the plugin is making a request for Neo4j connections. If none have been created yet, then the namespace does not yet exist in the metastore. When making a query to the metastore for a namespace which does not yet exist, working locally, this situation is handled a little more gracefully and does not result in an uncaught exception. When connected to a repository, a MetaStoreException is thrown. Currently, this exception is not being caught in the neo4j kettle plugin.
I downloaded the source code for both the neo4j-core and neo4j-output projects. With a little tweaking, including catching the MetaStoreException, as well as some additional null checks, I am able to get the neo4j plugin past that exception. I am able to create new Neo4j connections, as well as save the step and transformation to the repository. I can see the Neo4j namespace and connections in the repository metastore. I did not go as far as running the transformation, but that is enough to tell me that the issues with the Neo4j plugin and the metastore can be addressed with some of the measures I discussed above.

Many thanks for your help, those steps are much appreciated !!

JFM

@mattcasters

This comment has been minimized.

Copy link
Collaborator

commented Sep 20, 2019

Can you hit the "Details" button and paste the stack trace over here?
Thanks in advance!

@mattcasters

This comment has been minimized.

Copy link
Collaborator

commented Sep 20, 2019

In principle, none of the other MetaStore implementations throw exceptions if you ask for elements in a namespace that doesn't exist. You simply get an empty list back. Fortunately we already have a fork of the MetaStoreFactory in the Neo4j Core plugin project (because of bugs in the Pentaho version, a recurring theme) so we can fix it there.

@mattcasters mattcasters changed the title Unable to save a Neo4j connection using the PDI EE repository Unable to handle Neo4j connections with the PDI EE repository Sep 20, 2019
@mattcasters

This comment has been minimized.

Copy link
Collaborator

commented Sep 20, 2019

@jfmonteil

This comment has been minimized.

Copy link
Author

commented Sep 23, 2019

Hello, tested 4.1.1
No problem when opening the step while connected to the EE repository.
However the problem happens once I want to save a new connection :

org.pentaho.metastore.api.exceptions.MetaStoreException: Namespace 'Neo4j doesn't exist in the repository
at org.pentaho.di.repository.pur.metastore.PurRepositoryMetaStore.validateNamespace(PurRepositoryMetaStore.java:479)
at org.pentaho.di.repository.pur.metastore.PurRepositoryMetaStore.getElementTypes(PurRepositoryMetaStore.java:223)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.pentaho.di.ui.repo.timeout.MetaStoreSessionTimeoutHandler.invoke(MetaStoreSessionTimeoutHandler.java:45)
at com.sun.proxy.$Proxy194.getElementTypes(Unknown Source)
at org.pentaho.metastore.stores.delegate.DelegatingMetaStore.getElementTypes(DelegatingMetaStore.java:205)
at org.pentaho.metastore.stores.delegate.DelegatingMetaStore.getElementTypeByName(DelegatingMetaStore.java:236)
at org.neo4j.kettle.core.metastore.MetaStoreFactory.loadElement(MetaStoreFactory.java:78)
at bi.know.kettle.neo4j.shared.NeoConnectionUtils.newConnection(NeoConnectionUtils.java:39)
at bi.know.kettle.neo4j.steps.output.Neo4JOutputDialog.newConnection(Neo4JOutputDialog.java:1069)
at bi.know.kettle.neo4j.steps.output.Neo4JOutputDialog.access$000(Neo4JOutputDialog.java:57)
at bi.know.kettle.neo4j.steps.output.Neo4JOutputDialog$1.widgetSelected(Neo4JOutputDialog.java:176)
at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
at bi.know.kettle.neo4j.steps.output.Neo4JOutputDialog.open(Neo4JOutputDialog.java:718)
at org.pentaho.di.ui.spoon.delegates.SpoonStepsDelegate.editStep(SpoonStepsDelegate.java:120)
at org.pentaho.di.ui.spoon.Spoon.editStep(Spoon.java:8814)
at org.pentaho.di.ui.spoon.trans.TransGraph.editStep(TransGraph.java:3287)
at org.pentaho.di.ui.spoon.trans.TransGraph.mouseDoubleClick(TransGraph.java:785)
at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1384)
at org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:7949)
at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:9331)
at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:710)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.pentaho.commons.launcher.Launcher.main(Launcher.java:92)


> org.pentaho.metastore.api.exceptions.MetaStoreException: Namespace 'Neo4j doesn't exist in the repository
> 	at org.pentaho.di.repository.pur.metastore.PurRepositoryMetaStore.validateNamespace(PurRepositoryMetaStore.java:479)
> 	at org.pentaho.di.repository.pur.metastore.PurRepositoryMetaStore.getElementTypes(PurRepositoryMetaStore.java:223)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.pentaho.di.ui.repo.timeout.MetaStoreSessionTimeoutHandler.invoke(MetaStoreSessionTimeoutHandler.java:45)
> 	at com.sun.proxy.$Proxy194.getElementTypes(Unknown Source)
> 	at org.pentaho.metastore.stores.delegate.DelegatingMetaStore.getElementTypes(DelegatingMetaStore.java:205)
> 	at org.pentaho.metastore.stores.delegate.DelegatingMetaStore.getElementTypeByName(DelegatingMetaStore.java:236)
> 	at org.neo4j.kettle.core.metastore.MetaStoreFactory.loadElement(MetaStoreFactory.java:78)
> 	at bi.know.kettle.neo4j.shared.NeoConnectionUtils.newConnection(NeoConnectionUtils.java:39)
> 	at bi.know.kettle.neo4j.steps.output.Neo4JOutputDialog.newConnection(Neo4JOutputDialog.java:1069)
> 	at bi.know.kettle.neo4j.steps.output.Neo4JOutputDialog.access$000(Neo4JOutputDialog.java:57)
> 	at bi.know.kettle.neo4j.steps.output.Neo4JOutputDialog$1.widgetSelected(Neo4JOutputDialog.java:176)
> 	at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
> 	at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
> 	at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source)
> 	at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
> 	at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
> 	at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
> 	at bi.know.kettle.neo4j.steps.output.Neo4JOutputDialog.open(Neo4JOutputDialog.java:718)
> 	at org.pentaho.di.ui.spoon.delegates.SpoonStepsDelegate.editStep(SpoonStepsDelegate.java:120)
> 	at org.pentaho.di.ui.spoon.Spoon.editStep(Spoon.java:8814)
> 	at org.pentaho.di.ui.spoon.trans.TransGraph.editStep(TransGraph.java:3287)
> 	at org.pentaho.di.ui.spoon.trans.TransGraph.mouseDoubleClick(TransGraph.java:785)
> 	at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
> 	at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
> 	at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source)
> 	at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
> 	at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
> 	at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
> 	at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1384)
> 	at org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:7949)
> 	at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:9331)
> 	at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:710)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.pentaho.commons.launcher.Launcher.main(Launcher.java:92)
> 

@mattcasters

This comment has been minimized.

Copy link
Collaborator

commented Sep 23, 2019

OK, we get a bit further. Will harden more code and push out 4.1.2 then.

@mattcasters

This comment has been minimized.

Copy link
Collaborator

commented Sep 23, 2019

Try the new neo4j core library: https://github.com/mattcasters/kettle-neo4j-core/releases/tag/1.0.3
Replace version 1.0.2 in data-integration/plugins/Neo4JOutput/lib/
If that works I'll roll out a new version of the plugin so that the new core library is included.

@jfmonteil

This comment has been minimized.

Copy link
Author

commented Sep 24, 2019

GREAT Job !
It works fine now.

@jfmonteil jfmonteil closed this Sep 24, 2019
@mattcasters

This comment has been minimized.

Copy link
Collaborator

commented Sep 24, 2019

Thanks for the feedback JFM!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.