[ADD] add embedded BaseX as data source #79

Closed
wants to merge 1 commit into
from

Conversation

Projects
None yet
6 participants
@dirkk

dirkk commented Nov 23, 2013

The following pull requests adds the possibility to add embedded BaseX as a new data source for the runner. We are quite interested in not using XQJ to communicate with BaseX, for the following reasons:

  • we are limited to XQJ for the communication, i.e. future enhancements like debugging are incompatible with XGJ
  • by not using XQJ a user does not have to start an external BaseX server
  • XQJ is closed source and we do love open source!

We are looking into further enhancements (in general and with specific focus of BaseX) for this plugin, this is a first start. We are also interested in more input also from other vendors, so we don't want to break anything. In fact, I think it would also be interesting for others (e.g. exist-db) to use an embedded version and it should be quite easy to add this now.
Sorry about the not quite optimal pull request, I am very new to IntelliJ and IntelliJ plugin programming (and git wasn't really doing what I wanted it to do...). We are actually very happy about comments and improvements/suggestions/hints.

@buildhive

This comment has been minimized.

Show comment
Hide comment
@buildhive

buildhive Nov 23, 2013

ligasgr » intellij-xquery #30 FAILURE
Looks like there's a problem with this pull request
(what's this?)

ligasgr » intellij-xquery #30 FAILURE
Looks like there's a problem with this pull request
(what's this?)

@ligasgr

This comment has been minimized.

Show comment
Hide comment
@ligasgr

ligasgr Nov 23, 2013

Owner

Hi,

First of all, thanks for this contribution! I think it will be very useful!
About the usage of XQJ: it shouldn't make it much harder to introduce debugging but maybe I'm missing some important piece of information (although I must admit that using native runner would probably make it easier).
I started using XQJ because of the unified interface. Most of implementations of XQuery now come with XQJ interface.
Having to prepare a new runner for every implementation is not really something I'd like to see.
Maybe there could be a new implementation of XQJ created for BaseX that would allow using embedded BaseX (and that would be open source)?
Btw. does anyone know why XQJ implementations are closed source?

Here are a few things after having an initial look at the changes:

  1. Can you please rebase your branch against latest upstream master (maybe on Monday as there are some big changes coming in today and tomorrow)?
  2. Please remember about adding notice to newly created files.
  3. Please avoid adding unnecessary comments in the code if the code is self-explanatory (and it should be, if not please use better names).
  4. Please use try-finally wherever there's a possiblity of leaving stream/file not properly released.
  5. Why can't we re-use user defined library paths? There's already a logic for handling of all required cases and it's just a matter of changing label on the panel and it will be possible to avoid adding additional panels.
  6. Can you please use new property in XQueryDataSourceType for holding the Runner implementation class which will be used for given data source type with default value of XQueryXQJRunner? This will remove the need for nasty == comparisons. Based on the runner type decision can be made if xqj jars should be added or not.
  7. Can you please use wrapping margin of 120 characters instead of 80?
  8. There are some changes introduced that look accidental (enabling properties on gui forms, reformats with wrapping, insertions of new chars in some places, incorrect rebase). Can they be avoided? It will make the pull request more readable and will remove the need for manual merge).
  9. Can you please make sure that all tests are passing for your changes? (./gradlew clean test testGui)

Again, thanks a lot for raising this pull request!

Owner

ligasgr commented Nov 23, 2013

Hi,

First of all, thanks for this contribution! I think it will be very useful!
About the usage of XQJ: it shouldn't make it much harder to introduce debugging but maybe I'm missing some important piece of information (although I must admit that using native runner would probably make it easier).
I started using XQJ because of the unified interface. Most of implementations of XQuery now come with XQJ interface.
Having to prepare a new runner for every implementation is not really something I'd like to see.
Maybe there could be a new implementation of XQJ created for BaseX that would allow using embedded BaseX (and that would be open source)?
Btw. does anyone know why XQJ implementations are closed source?

Here are a few things after having an initial look at the changes:

  1. Can you please rebase your branch against latest upstream master (maybe on Monday as there are some big changes coming in today and tomorrow)?
  2. Please remember about adding notice to newly created files.
  3. Please avoid adding unnecessary comments in the code if the code is self-explanatory (and it should be, if not please use better names).
  4. Please use try-finally wherever there's a possiblity of leaving stream/file not properly released.
  5. Why can't we re-use user defined library paths? There's already a logic for handling of all required cases and it's just a matter of changing label on the panel and it will be possible to avoid adding additional panels.
  6. Can you please use new property in XQueryDataSourceType for holding the Runner implementation class which will be used for given data source type with default value of XQueryXQJRunner? This will remove the need for nasty == comparisons. Based on the runner type decision can be made if xqj jars should be added or not.
  7. Can you please use wrapping margin of 120 characters instead of 80?
  8. There are some changes introduced that look accidental (enabling properties on gui forms, reformats with wrapping, insertions of new chars in some places, incorrect rebase). Can they be avoided? It will make the pull request more readable and will remove the need for manual merge).
  9. Can you please make sure that all tests are passing for your changes? (./gradlew clean test testGui)

Again, thanks a lot for raising this pull request!

@fancellu

This comment has been minimized.

Show comment
Hide comment
@fancellu

fancellu Nov 24, 2013

I would avoid going to custom drivers, this way lies madness. As it is, I hear that Charles Foster, the owner of XQJ.net and author of the XQJ for BaseX drivers (amongst others) is working on an embedded BaseX driver for release in the next few days. As for it being closed source, well, its his code. If someone wants an open source version that much, you are all free to create your own. Be grateful that he's provided this at all! You can always ask for your money back.

I would avoid going to custom drivers, this way lies madness. As it is, I hear that Charles Foster, the owner of XQJ.net and author of the XQJ for BaseX drivers (amongst others) is working on an embedded BaseX driver for release in the next few days. As for it being closed source, well, its his code. If someone wants an open source version that much, you are all free to create your own. Be grateful that he's provided this at all! You can always ask for your money back.

@ChristianGruen

This comment has been minimized.

Show comment
Hide comment
@ChristianGruen

ChristianGruen Nov 24, 2013

I’m really interested to see how/if XQJ can be used to provide support for all the implementation-defined features of available XQuery implementations. One example: BaseX has a repository directory in which all modules are located that are not further specified by a import module location. The repo location is stored in a configuration directory (more details). As this directory is currently not known to the XQuery plugin, features like context sensitive code completion don’t work yet.

I’m really interested to see how/if XQJ can be used to provide support for all the implementation-defined features of available XQuery implementations. One example: BaseX has a repository directory in which all modules are located that are not further specified by a import module location. The repo location is stored in a configuration directory (more details). As this directory is currently not known to the XQuery plugin, features like context sensitive code completion don’t work yet.

@cfoster

This comment has been minimized.

Show comment
Hide comment
@cfoster

cfoster Nov 25, 2013

@ChristianGruen Regardless of whether you used XQuery, commands, or some kind of API, I think this would be quite tricky. Each XQuery Processor has different ways of approaching things, like each XML Database approaches databases, storage, collections and XQuery module storage to some extent differently.
One answer might be finding common ground between processors, then defining some kind of standard XQuery interface to inspect and change implementation defined features.

cfoster commented Nov 25, 2013

@ChristianGruen Regardless of whether you used XQuery, commands, or some kind of API, I think this would be quite tricky. Each XQuery Processor has different ways of approaching things, like each XML Database approaches databases, storage, collections and XQuery module storage to some extent differently.
One answer might be finding common ground between processors, then defining some kind of standard XQuery interface to inspect and change implementation defined features.

@dirkk

This comment has been minimized.

Show comment
Hide comment
@dirkk

dirkk Nov 25, 2013

@ligasgr Sure thing I will improve all the coding quality and code style things you did mention.

Regarding the general discussion native drivers vs. XQJ: I totally understand why you choose XQJ and it certainly makes sense to use a unified way of access. However, as Christian mentioned we can't really imagine how to make many things work using XQJ as lots of things are actually implementation-dependent. I totally agree with @cfoster that we should find as much common ground as possible, but after having a general architecture I think we need some implementation-specific operations. Having a standard XQuery interface sounds appealing, but I am not sure it can really cover everything and I am also not convinced that many vendors would be interested.
Having an XQJ driver for embedded BaseX would certainly be nice, but we are not aware of any planned release. @cfoster, could you say something about this?

In general, I don't see much madness in using native drivers. Actually, the implementation of accessing BaseX is quite small and I don't think that there is going on much change in the API for the set-up - and I expect the same for all the other processors.

And I don't think a discussion closed source vs. open source is of much use here and rather cumbersome. But still we should be allowed to criticize and scrutinize the fact, that an OS language Plugin is talking to an OS processor/database and in between is something closed source.

dirkk commented Nov 25, 2013

@ligasgr Sure thing I will improve all the coding quality and code style things you did mention.

Regarding the general discussion native drivers vs. XQJ: I totally understand why you choose XQJ and it certainly makes sense to use a unified way of access. However, as Christian mentioned we can't really imagine how to make many things work using XQJ as lots of things are actually implementation-dependent. I totally agree with @cfoster that we should find as much common ground as possible, but after having a general architecture I think we need some implementation-specific operations. Having a standard XQuery interface sounds appealing, but I am not sure it can really cover everything and I am also not convinced that many vendors would be interested.
Having an XQJ driver for embedded BaseX would certainly be nice, but we are not aware of any planned release. @cfoster, could you say something about this?

In general, I don't see much madness in using native drivers. Actually, the implementation of accessing BaseX is quite small and I don't think that there is going on much change in the API for the set-up - and I expect the same for all the other processors.

And I don't think a discussion closed source vs. open source is of much use here and rather cumbersome. But still we should be allowed to criticize and scrutinize the fact, that an OS language Plugin is talking to an OS processor/database and in between is something closed source.

@cfoster

This comment has been minimized.

Show comment
Hide comment
@cfoster

cfoster Nov 25, 2013

I could release BaseX XQJ 1.3 today, which includes a local / embedded XQDataSource. However, it uses the LocalQuery interface to talk to BaseX. For performance reasons, I understand that this might not be the best approach, I am as much of a fan of performance as I am of standards. So I'm considering another approach (still discussing with Christian).

I agree with @fancellu that using proprietary interfaces to execute XQuery expressions instead of a standard interface like XQJ is madness, regardless of how small the proprietary interface you use to execute XQuery expressions is. For example, I originally created 99% of BaseX XQJ 1.3 (embedded) when BaseX was at version 7.5. Recently, coming back to BaseX XQJ, I found that the BaseX LocalQuery and related signatures had changed from BaseX 7.5 to 7.6 and as a result my code couldn't compile anymore. The BaseX signatures also changed slightly from 7.6 to 7.7. If I had of released BaseX XQJ 1.3 at BaseX 7.5, the compiled embedded driver would have stopped working with newer versions of BaseX because compiled method signatures would be incompatible. The same would be true of any other compiled software which used those original interfaces.

Standards are very important. XQJ, like XQuery, is a standard. Take away standards and there is likely to be chaos.

With a view to building extensions, there are some successful things out there that have built on top of of JDBC and SQL Databases, like Hibernate for instance. Certainly for the XQuery/XPath world, EXPath and EXQuery are promising steps forward. I think there should be more of a focus on an XQuery interface to provide common ground for configuring and inspecting things -- then we don't even need a Java interface! But this is not entirely straight-forward, on this configuration side of things, take MarkLogic for instance, it's admin console to configure App Servers, Databases and Storage is huge and complex --- they also provide an XQuery API to inspect and configure things, take a look at: http://docs.marklogic.com/admin

Quite a lot to take on board.

On another note, the BaseX Maven build has BaseX XQJ as a dependency, I have even been thinking about the argument of putting some XQJ tests in the BaseX build to ensure that the current BaseX XQJ embedded driver definitely works against the current BaseX code being built.

cfoster commented Nov 25, 2013

I could release BaseX XQJ 1.3 today, which includes a local / embedded XQDataSource. However, it uses the LocalQuery interface to talk to BaseX. For performance reasons, I understand that this might not be the best approach, I am as much of a fan of performance as I am of standards. So I'm considering another approach (still discussing with Christian).

I agree with @fancellu that using proprietary interfaces to execute XQuery expressions instead of a standard interface like XQJ is madness, regardless of how small the proprietary interface you use to execute XQuery expressions is. For example, I originally created 99% of BaseX XQJ 1.3 (embedded) when BaseX was at version 7.5. Recently, coming back to BaseX XQJ, I found that the BaseX LocalQuery and related signatures had changed from BaseX 7.5 to 7.6 and as a result my code couldn't compile anymore. The BaseX signatures also changed slightly from 7.6 to 7.7. If I had of released BaseX XQJ 1.3 at BaseX 7.5, the compiled embedded driver would have stopped working with newer versions of BaseX because compiled method signatures would be incompatible. The same would be true of any other compiled software which used those original interfaces.

Standards are very important. XQJ, like XQuery, is a standard. Take away standards and there is likely to be chaos.

With a view to building extensions, there are some successful things out there that have built on top of of JDBC and SQL Databases, like Hibernate for instance. Certainly for the XQuery/XPath world, EXPath and EXQuery are promising steps forward. I think there should be more of a focus on an XQuery interface to provide common ground for configuring and inspecting things -- then we don't even need a Java interface! But this is not entirely straight-forward, on this configuration side of things, take MarkLogic for instance, it's admin console to configure App Servers, Databases and Storage is huge and complex --- they also provide an XQuery API to inspect and configure things, take a look at: http://docs.marklogic.com/admin

Quite a lot to take on board.

On another note, the BaseX Maven build has BaseX XQJ as a dependency, I have even been thinking about the argument of putting some XQJ tests in the BaseX build to ensure that the current BaseX XQJ embedded driver definitely works against the current BaseX code being built.

@ChristianGruen

This comment has been minimized.

Show comment
Hide comment
@ChristianGruen

ChristianGruen Nov 25, 2013

Well, I have no doubt that, if proprietary drivers are used in practice, there are enough reasons for keeping the used APIs more stable, or (just in case) that there will always be people who will update the interface if necessary.

The obvious reason for implementation-specific features and drivers is that existing standards don’t provide all required features. Maybe we could start a list of non-standard features that are currently not supported by XQJ or the XQuery Plugin? This way, we could find out more easily if it’s realistic to get standardized support for such features in a foreseeable time frame.

@cfoster: I agree that JUnit tests for the BaseX ↔ XQJ dependency are quite essential; otherwise, we’ll run into the very same problems as you described above.

Well, I have no doubt that, if proprietary drivers are used in practice, there are enough reasons for keeping the used APIs more stable, or (just in case) that there will always be people who will update the interface if necessary.

The obvious reason for implementation-specific features and drivers is that existing standards don’t provide all required features. Maybe we could start a list of non-standard features that are currently not supported by XQJ or the XQuery Plugin? This way, we could find out more easily if it’s realistic to get standardized support for such features in a foreseeable time frame.

@cfoster: I agree that JUnit tests for the BaseX ↔ XQJ dependency are quite essential; otherwise, we’ll run into the very same problems as you described above.

@dirkk

This comment has been minimized.

Show comment
Hide comment
@dirkk

dirkk Nov 25, 2013

After some more discussion with Christian I have some further points to mention: First of all, I really don't care much about having a native driver here or this pull request in particular. Actually, I just really like to have things working, so this is key here. If everything works fine using XQJ it is even better, because it also simplifies things for us (i.e. we don't have to maintain BaseX specific code for this plugin).

So actually, for now I would be also quite happy with an embedded BaseX XQJ solution (using the server version looks quite impractical to me and I think it should be as easy to use as possible).
Let me also add that we are certainly also a fan of standards and we (i.e. actually Christian is as he is editor and contributor of several EXPath modules...) support EXPath and all standarization efforts going on. But as much as this is promising, there are still lots of differences between the different processors and I don't think they are likely to go away very fast.

So we are planning to collecting issues which we think are difficult to tackle using the current XQJ driver during our development for this plugin. If we then discover that it is simply to difficult or time-consuming to standardize this, maybe the positive sites of using a native driver outweigh the negative ones (certainly it is more effort to maintain a native driver).

dirkk commented Nov 25, 2013

After some more discussion with Christian I have some further points to mention: First of all, I really don't care much about having a native driver here or this pull request in particular. Actually, I just really like to have things working, so this is key here. If everything works fine using XQJ it is even better, because it also simplifies things for us (i.e. we don't have to maintain BaseX specific code for this plugin).

So actually, for now I would be also quite happy with an embedded BaseX XQJ solution (using the server version looks quite impractical to me and I think it should be as easy to use as possible).
Let me also add that we are certainly also a fan of standards and we (i.e. actually Christian is as he is editor and contributor of several EXPath modules...) support EXPath and all standarization efforts going on. But as much as this is promising, there are still lots of differences between the different processors and I don't think they are likely to go away very fast.

So we are planning to collecting issues which we think are difficult to tackle using the current XQJ driver during our development for this plugin. If we then discover that it is simply to difficult or time-consuming to standardize this, maybe the positive sites of using a native driver outweigh the negative ones (certainly it is more effort to maintain a native driver).

@cfoster

This comment has been minimized.

Show comment
Hide comment
@cfoster

cfoster Nov 26, 2013

Releasing BaseX XQJ 1.3 with a local / embedded XQDataSource is my highest priority at the moment and should be released within the next few days. For those wanting to keep an eye on the progress and how it interfaces with BaseX directly, I've put a GitHub repo here:

https://github.com/cfoster/basex-xqj-local

cfoster commented Nov 26, 2013

Releasing BaseX XQJ 1.3 with a local / embedded XQDataSource is my highest priority at the moment and should be released within the next few days. For those wanting to keep an eye on the progress and how it interfaces with BaseX directly, I've put a GitHub repo here:

https://github.com/cfoster/basex-xqj-local

@ligasgr

This comment has been minimized.

Show comment
Hide comment
@ligasgr

ligasgr Dec 2, 2013

Owner

Hi,

To sum up few comments from my side:

  1. I don't feel overly tied to XQJ. If it'll prove to not give the expected flexibility/required feature set I won't hesitate much and will replace it where needed.
  2. @ChristianGruen - this repository feature (the referencing/autocompletion part) is not in any way related to XQJ. It will be a separate feature that will be vendor independent to add the posibility to scan through external sources. iI think that passing in location of repository for the runnable configuration is already possible via system properties (on Java Configuration tab when configuring run).
  3. @dirkk - can you also please add tests where possible?
  4. @cfoster - thanks for taking a look at releasing the embedded version of BaseX xqj driver!
  5. Thanks all for your contribution! Hope to see new pull request soon!

Cheers!

Owner

ligasgr commented Dec 2, 2013

Hi,

To sum up few comments from my side:

  1. I don't feel overly tied to XQJ. If it'll prove to not give the expected flexibility/required feature set I won't hesitate much and will replace it where needed.
  2. @ChristianGruen - this repository feature (the referencing/autocompletion part) is not in any way related to XQJ. It will be a separate feature that will be vendor independent to add the posibility to scan through external sources. iI think that passing in location of repository for the runnable configuration is already possible via system properties (on Java Configuration tab when configuring run).
  3. @dirkk - can you also please add tests where possible?
  4. @cfoster - thanks for taking a look at releasing the embedded version of BaseX xqj driver!
  5. Thanks all for your contribution! Hope to see new pull request soon!

Cheers!

@cfoster

This comment has been minimized.

Show comment
Hide comment
@cfoster

cfoster Dec 4, 2013

I have created BaseX XQJ 1.3.0-SNAPSHOT which has a local / embedded XQDataSource that passes Oracle's XQJ TCK. I've also made a small Gist to help anyone who wants to try it out [1].

This SNAPSHOT is essentially a beta and should not be used in a production environment. Once I have carried out more testing and added tests into the BaseX build and Christian is happy, then BaseX XQJ 1.3.0 final should be ready to ship.

In theory, this should work just fine with the IntelliJ XQuery Plugin.

[1] https://gist.github.com/cfoster/7793034

cfoster commented Dec 4, 2013

I have created BaseX XQJ 1.3.0-SNAPSHOT which has a local / embedded XQDataSource that passes Oracle's XQJ TCK. I've also made a small Gist to help anyone who wants to try it out [1].

This SNAPSHOT is essentially a beta and should not be used in a production environment. Once I have carried out more testing and added tests into the BaseX build and Christian is happy, then BaseX XQJ 1.3.0 final should be ready to ship.

In theory, this should work just fine with the IntelliJ XQuery Plugin.

[1] https://gist.github.com/cfoster/7793034

@ChristianGruen

This comment has been minimized.

Show comment
Hide comment
@ChristianGruen

ChristianGruen Dec 4, 2013

Charles, thanks so much for your efforts!

Charles, thanks so much for your efforts!

@dirkk

This comment has been minimized.

Show comment
Hide comment
@dirkk

dirkk Dec 5, 2013

I'll be closing this pull request as I think at least at the moment the embedded XQJ BaseX version is satisfying our needs. By doing so I hope we are avoiding this nativ vs. XQJ driver discussion, at least for the moment and can focus on more important enhancement for this module.
Thanks @cfoster for implementing this!

I think I'll be sending another pull request for an embedded BaseX version using Charles' XQJ driver. This time I am making sure to include tests and remove comments (by the way, that's a request I have never gotten before... normally people complain that I comment not enough :-) )

dirkk commented Dec 5, 2013

I'll be closing this pull request as I think at least at the moment the embedded XQJ BaseX version is satisfying our needs. By doing so I hope we are avoiding this nativ vs. XQJ driver discussion, at least for the moment and can focus on more important enhancement for this module.
Thanks @cfoster for implementing this!

I think I'll be sending another pull request for an embedded BaseX version using Charles' XQJ driver. This time I am making sure to include tests and remove comments (by the way, that's a request I have never gotten before... normally people complain that I comment not enough :-) )

@dirkk dirkk closed this Dec 5, 2013

ligasgr added a commit that referenced this pull request Mar 18, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment