Remove XQueryServlet

[adamretter]

For quite some time XQueryServlet has been deprecated and we have recommended that people instead store their XQuery in the database. The eXist book also makes the same recomendation.
I would like to remove this old for a long time deprecated code.

[dizzzz]

Isn't the controller using this servlet? or betterForm? It would cause regression for some people, but for a v3.0 release -when correctly documented and announced- acceptible IMO.

[shabanovd]

It used for in db xquery scripts too. Maybe, it can be split by usecases: db, FS and code as request parameter. I'd like to remove last one, but there people that use it, so removing may/will break there apps.

[wolfgangmm]

It is still being used quite a lot as I know from customers, e.g. in applications which embed eXist. In some scenarios it makes sense to keep queries on the file system.

eXide also uses it to post XQuery code if you press the "Eval" button (same for the demo app). Changing all this would imply a lot of work, taking time from other things.

[adamretter]

@shabanovd Sorry can you tell me why it would be used in DB XQuery Scripts? I thought that was what RESTServer was for?

@wolfgangmm Can you explain why eXide uses this rather than posting the XQuery via RESTServer?

[dizzzz]

@adamretter please read wolf's answer a bit slower ; there is no real reason as I read it but since there are dependencies with for example eXide, and timing constraints, it cannot be removed.

[adamretter]

@dizzzz I read the answer. I just wanted more information. Surely I could do the changes on eXide if I wished to - I think it is still Open Source right?

[dizzzz]

:-) yes :-)

[wolfgangmm]

Ignore eXide. We can change that ourselves if we like to.

The relevant info is that some of the biggest users of eXist I know of still rely on XQueryServlet in an embedded context.

[adamretter]

@wolfgangmm Is the goal to continue to advise against the XQueryServlet. Is so what is the timescale for removing it?

[JoernT]

I think it was not wrong what we did with the SOAP API - we can simply ask people if they still rely on it (and why) and probably (optionally) make clear that the XQueryServlet is deprecated and give some advise what to use instead. If there are still users of it we can probably set some relaxed deadline before removal.

However if there's still a valid use case for it - XQueries being stored on the filesystem or similar - then we should probably rethink the deprecation.

[wolfgangmm]

@adamretter I always advise against the XQueryServlet and communicate this to our customers. Some of them are currently struggling hard (involving large teams) to finally migrate to eXist 2.x and I certainly don't want to make it even harder for them by raising the bar too high. Switching from a file system based architecture to one in the database is not always straightforward. We definitely have to give those guys more time to adopt (end of the year I'd say).

[ljo]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Den 2015-04-02 14:01, Wolfgang Meier skrev:

definitely have to give those guys more time to adopt (end of the
year I'd say).

Yes, I suggest we decide to communicate a date in the coming release.
So the next next (the one after the coming) release would be an
obvious target I say.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iD8DBQFVHTE9hcIn5aVXOPIRAvqtAJ9Gy09j/Gjjv7sI/5FC2iQ6Q9vfawCgkYUA
7OHII33d4Uwgu062QGl7QpE=
=XTQv
-----END PGP SIGNATURE-----

[JoernT]

ok, i just learned that there are defnitely people using it and migration would cause a lot of work. So yes, we need a relaxed timespan for them to migrate. We even might need to show the way.

[adamretter]

Okay scheduled for eXist 4.0.

[ssire]

I use eXist-DB with most of my code (XQuery / XSLT) on the file system (in a version controlled folder under the webapp folder). I find it very convenient as it supports a clean workflow using a single "git pull origin master" command to deploy applications on production servers. I also develop with a file system editor (Text Mate) which I guess is common for many developers.

With the projected change and code restricted to the database only I fear of loosing some flexibility as if I understand the only options will be to use the embedded eXide IDE ? And then how to work under version control ? If I remember there is the possibility to mirror the code on the file system and use a synchronize command in eXide ? But on a production server, if we remove eXide, does that mean that all deployments should be made via the package manager ? I feel that I will regret the simple "git pull origin master" to update a server...

[adamretter]

@ssire Okay don't worry. eXist 4.0 is a long way away and we may keep indeed keep it if people like yourself are using it and we don't have a suitable replacement for version control

[joewiz]

eXist 4.0 is now the next major release. If removing the XQueryServlet is still under consideration, now may be the time to begin planning and determining how to minimize the impact on the valid use cases. @wolfgangmm mentioned those who still "rely on XQueryServlet in an embedded context." @JoernT mentioned "XQueries being stored on the filesystem" (e.g., @ssire's desire to be able to update a server with git pull origin master) and cited the example of how the SOAP API was removed as a model.

[dizzzz]

I remember we still get feedback from users that have queries stored on the filesystem.....

If we have the removal on 'the roadmap' , it is OK for me; Maybe (...) we can disable the feature initially (so ppl can provide feedback on this) en then remove it entirely....

[shabanovd]

I think that 1st step should be to disable it by default, so people that need it can enable it.

[caseydawsonjordan]

As a data point, we will always need something like the xquery servlet as we cannot store all of our xqueries in the database, and even if we could I believe it would be a very difficult reactor. What is the reason behind not using the servlet? I believe there are legitimate reasons why people would not store xquery in the database, seems reasonable that the servlet should exist. Thanks

…

On Tue, Feb 21, 2017, 6:04 AM Dmitriy Shabanov ***@***.***> wrote: I think that 1st step should be to disable it by default, so people that need it can enable it. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#555 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAgl3ewFooKsYRJYx7qaLOB8IfqYhd6zks5resTLgaJpZM4D5AdU> .

[ssire]

IMHO the ability to use "git pull" out of the box for production code deployment (and for code versioning) is what makes eXist-DB as friendly as other popular web applications development frameworks and agile. Having to add an extra step to synchronize file system with the the database sounds feasible but will raise the maintenance costs and is prone to errors.

By the way how redundant is the XQueryServlet code with the code that execute XQuery resources stored in the database ?

[adamretter]

So the XQueryServlet likely has a whole bunch of security issues. Including the fact that there are no users or permissions available from executing an XQuery from the filesystem via XQueryServlet. Effectively anyone can execute anything... that is very bad.

[caseydawsonjordan]

True. But this is the same for pretty much all interpreted languages and is common to develop that way. The developer is responsible for security in that case, and that is very reasonable. Imo having the ability for files in the db to be executed is a much larger security problem, since an attacker has many more opportunities to create a malicious file that can be executed over the filesystem. Getting permissions inside the db correct is much more complex imo and I would wager most probably don't do it right.

…

On Tue, Feb 21, 2017, 12:19 PM Adam Retter ***@***.***> wrote: So the XQueryServlet likely has a whole bunch of security issues. Including the fact that there are no users or permissions available from executing an XQuery from the filesystem via XQueryServlet. Effectively anyone can execute anything... that is very bad. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#555 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAgl3RMi0gBVd0mcSYytxkpvYDZKu6Qpks5rexytgaJpZM4D5AdU> .

[joewiz]

An issue tracking the documentation implications of this PR has been started: eXist-db/documentation#569.