cider-uruk-any should be more helpful with errors

[hanshuebner]

Right now, all exceptions are just printed using the standard cider exception handler:

  Show: Clojure Java REPL Tooling Duplicates All  (11 frames hidden)

1. Unhandled com.marklogic.xcc.exceptions.XQueryException
   com.marklogic.xcc.exceptions.XQueryException: XDMP-ARGTYPE:
   (err:XPTY0004)
   fn:substring((fn:doc("/tx-mpf/1597783.xml")/group/provider[1]/provider-group-membership[1]/group-tpisuffix-code,
   fn:doc("/tx-mpf/1597783.xml")/group/provider[1]/provider-group-membership[2]/group-tpisuffix-code),
   1, 7) -- arg1 is not of type xs:string?  [Session: user=restuser,
   cb=tx-mpf [ContentSource: user={none}, cb={none} [provider:
   address=localhost/127.0.0.1:8000, pool=1/64]]] [Client: XCC/8.0-5,
   Server: XDBC/8.0-4.2] in /eval, on line 19 expr:
   fn:substring((fn:doc("/tx-mpf/1597783.xml")/group/provider[1]/provider-group-membership[1]/group-tpisuffix-code,
   fn:doc("/tx-mpf/1597783.xml")/group/provider[1]/provider-group-membership[2]/group-tpisuffix-code),
   1, 7), in
   local:validate-with-tpi(fn:doc("/claims/10150330.xml")/c:claim/c:access-point/c:medicaid-id/text(),
   fn:doc("/claims/10150330.xml")/c:claim) in /eval, on line 63 expr:
   fn:substring((fn:doc("/tx-mpf/1597783.xml")/group/provider[1]/provider-group-membership[1]/group-tpisuffix-code,
   fn:doc("/tx-mpf/1597783.xml")/group/provider[1]/provider-group-membership[2]/group-tpisuffix-code),
   1, 7)

ServerExceptionHandler.java:   34  com.marklogic.xcc.impl.handlers.ServerExceptionHandler/handleResponse
EvalRequestController.java:   96  com.marklogic.xcc.impl.handlers.EvalRequestController/serverDialog

It would be nice if the XQuery stacktrace was displayed instead. and if there was an easy way to navigate to the XQuery error location from there.

[proofit404]

If in /eval, on line 19 expr part of xquery stacktrace pointed to the right document location, it is possible to write compilation-minor-mode compatible regex. It's possible to navigate with next-error and previous-error keys.

Is it what are you looking for?

[hanshuebner]

Exactly, the error location in the XQuery evaluator stack trace is accurate and currently, I'm locating the error manually by looking at the on line 63 expr lines. Also, it would be good if the XQuery stack traces would be alone, not mixed with the Clojure stack traces. If that is possible at all.

[proofit404]

In the branch error-buffer following behavior is implemented:

• cider repl is connected
• you select any xquery buffer, for testing purpose let say we have following buffer content

xquery version "1.0-ml";
module namespace hello = "helloworld";

declare function helloworld()
{
  <foo>
};

• you type C-c C-c to evaluate whole buffer
• errors buffer pops up, this is its content

XQueryException com.marklogic.xcc.exceptions.XQueryException: XDMP-BADCHAR: (err:XPST0003) Unexpected character found '}' (0x007d)
 [Session: user=proofit404, cb=TutorialDB [ContentSource: user=proofit404, cb=TutorialDB [provider: address=localhost/127.0.0.1:8889, pool=1/64]]]
 [Client: XCC/8.0-5, Server: XDBC/8.0-5.8]
on line 7
expr:    com.marklogic.xcc.impl.handlers.ServerExceptionHandler.handleResponse (ServerExceptionHandler.java:34)

• on line 7 phrase is highlighted
• you can call next-error function to focus this line in the origin xquery buffer (bound to C-x ` by default)

Now things becomes more interesting when we speak about errors inside documents uploaded to the MarkLogic server. For example syntax error can be placed inside module on the other server. In my installation I store xquery modules in the /data/ directory on the remote server. Lets upload xquery module from previous example to the /data/e/test.xqy to the marklogic host.

Now we open another xquery file localy

• lets say our next buffer has this content

xquery version "1.0-ml";
import module namespace hw="helloworld" at "/e/test.xqy";

hw:helloworld()

• we press C-c C-c again to evaluate it
• error buffer pops up with following message

XQueryException com.marklogic.xcc.exceptions.XQueryException: XDMP-BADCHAR: (err:XPST0003) Unexpected character found '}' (0x007d)
 [Session: user=proofit404, cb=TutorialDB [ContentSource: user=proofit404, cb=TutorialDB [provider: address=localhost/127.0.0.1:8889, pool=1/64]]]
 [Client: XCC/8.0-5, Server: XDBC/8.0-5.8]
in /e/test.xqy, on line 7
expr:    com.marklogic.xcc.impl.handlers.ServerExceptionHandler.handleResponse (ServerExceptionHandler.java:34)

• in /e/test.xqy, on line 7 phrase is highlighted
• if we press C-x ` we will see content of module hosted on the server

Please give it a try and let me know if any this is missing in this feature.

[m-g-r]

Hi, this starts to look nice! For now, there seem to be some problems.

• Sometimes after "expr: " there is no expression mentioned. Most of the times the problematic expression is mentioned, but often not (as in your paste above, just: "expr: com.marklog[...]"
• I have error cases where I just get a completely empty buffer and have no means to look at the error. Maybe there should be a means to retrieve the full Java stacktrace as well, because otherwise it sometimes gives you no information at all as it is now.
  The case was a bit more complicated with a module that contained an explicit fn:error. I tried to come up with a simple error case but sadly all error cases I came up with correctly show a XQuery-Error... buffer with the message. Sorry.
• The opening of file location works for local files, quite nice! But I cannot confirm that it works on uploaded modules. Looking at the code, it seems you expect to be the modules database to be the same as the content base? That is not true in general, and also not for us. So, in general, you would have to ask the MarkLogic about the current database's modules database.
  I do that in xdbc-selector already, have a look at xdmp-get-modules-database in https://github.com/xquery-mode/xdbc-selector/blob/master/xdmp-methods.el. With that and also already with cider-any-uruk-document-get we start to have duplicated functionality. I agree that it would be nice to access the code from cider-any-uruk. Would it be better to move those functionality from xdbc-selector to cider-any?

But all together, this look promising. Good work! Maybe make the messages more readable by moving less interesting information down. Have a look at:

XQueryException com.marklogic.xcc.exceptions.XQueryException: XDMP-NOTANODE: (err:XPTY0019) 23 | 23 -- 23 is not a node
 [Session: user=admin, cb={default} [ContentSource: user=admin, cb={none} [provider: address=localhost/127.0.0.1:8021, pool=1/64]]]
 [Client: XCC/8.0-5, Server: XDBC/8.0-5.5]
on line 4
expr: 23 | 23  com.marklogic.xcc.impl.handlers.ServerExceptionHandler.handleResponse (ServerExceptionHandler.java:34)

The interesting part only starts after "XQueryException com.marklogic.xcc.exceptions.XQueryException: ", and the [Session...][Client...] lines cold be moved below the error message itself, right?

Bye,
Max

[proofit404]

Hi!

First of all I want to describe how it was implemented.

We have cider-any-eval-handler which will take error reported by clojure uruk library. Error here is simply a string with traceback.

We show error buffer in any case without additional preprocessing.

Then if there is lines like "on line <num>" or "in <doc>, on line <num>" we will treat them as links to documents and support next or previous errors navigation.

Pretty simple, right?

Now let me explain each item in your list:

1. If there is no helpful information after expr: ..., that mean uruk library report it this way and there is nothing we can do about it. Error navigation based of line marker not expr anyway.
2. I will try to failback to previous behavior, if error was an empty string. But I'm not sure cider will process it correctly, since it use same error value as my handler do.

2.1. Please, submit reproducible document if possible. You can skip it minimization.

1. I will read xdmp:modules-database() documentation careful, hope it is possible to use it in remote modules navigation.

3.1. I don't think we need to move xdbc logic into cider-any-uruk. At least until we finish this branch.

Finally on error content modification. I think if we would modify it, we must do it in the clojure layer, when exception object is accessible. Not in Emacs Lisp when it only a string with new lines. I'll try to figure out if it possible based on current uruk code base.

Regards, Artem.

[proofit404]

Hello guys, I'm stuck :(

Expected usage

I'm trying to implement content access for MarkLogic modules. While navigate in the error buffer I want to see the actual position in the module, which triggers error. For example we have this file:

xquery version "1.0-ml";
import module namespace sec="http://marklogic.com/xdmp/security" at "/MarkLogic/security.xqy";

sec:user-get-roles("Jim")

Now I type C-c C-c. Following error buffer shows up:

XQueryException com.marklogic.xcc.exceptions.XQueryException: SEC-USERDNE: (err:FOER0000) User does not exist: sec:user-name = Jim
 [Session: user=proofit404, cb=TutorialDB [ContentSource: user=proofit404, cb=TutorialDB [provider: address=localhost/127.0.0.1:8889, pool=1/64]]]
 [Client: XCC/8.0-5, Server: XDBC/8.0-5.8]
in /MarkLogic/security.xqy, on line 3612
expr:  ,
in get-element("http://marklogic.com/xdmp/security", "sec:user", "sec:user-name", "Jim", "SEC-USERDNE")
in /MarkLogic/security.xqy, on line 1265
expr:  ,
in sec:user-get-roles("Jim")
on line 4
expr:    com.marklogic.xcc.impl.handlers.ServerExceptionHandler.handleResponse (ServerExceptionHandler.java:34)

We see three frames of traceback for this error. The first one stays in the /MarkLogic/security.xqy module. When I call next-error command cursor will stays at front of the first in ... expression and tries to open security module. This module stored in the modules database and I can obtain it with this xquery:

xdmp:document-get("/opt/MarkLogic/Modules/MarkLogic/security.xqy")

But how do I know this full path to the module? I try to google this solution for two days with out any success...

Then when I call next-error one more time I expect cursor moves its position in the security buffer from 3612 to the 1265 line.

After next call to next-error command we will stay inside our file in the line 4.

All this thing is working except I can't figure out how to obtain module content from module uri.

Modules database setting

Also there are two possible scenarios according to "modules database" setting. It is related to modules which were uploaded ourselves. If we set "modules database" setting to the "(file system)", our modules will be loaded from file system root. To resolve those modules we need something like

if (xdmp:modules-database() = 0)
then
  xdmp:document-get(fn:concat(xdmp:modules-root(), "/MarkLogic/security.xqy"))
else
  ...

But in general "modules database" setting will be set to something like "Modules" so our modules will be uploaded to this database too. I've tried to do something like this:

declare variable $module :=
  <module>
    <uri>/MarkLogic/security.xqy</uri>
  </module>;

let $mdb := xdmp:modules-database()

xdmp:eval(fn:concat('xdmp:document-get("', $module/uri, '")'),
          (),
          <options xmlns="xdmp:eval">
            <database>{$mdb}</database>
          </options>)

But it doesn't do what I expect.

Any suggestions?

Regards, Artem.

[m-g-r]

Hi Artem,

Short answer: the Modules of the Marklogic installation itself are special. Try with a user Module that you've added yourself; it should work for that case.

Please have another look at how MarkLogic resolves paths on module loading:
https://docs.marklogic.com/guide/app-dev/import_modules#id_29407

• An app server has modules. Either in a modules database or in the filesystem. Regardless of filesystem or database, the xdmp:modules-root() is obeyed.
• But also there is the Modules directory of the Marklogic Installation itself, usually in MARKLOGIC_INSTALL_DIR/Modules (see /etc/sysconfig/MarkLogic in Linux). And this Marklogic Modules directory is always asked first, and it isa filesystem location.

The problem now are:

• /MarkLogic/security.xqy is part of MarkLogic and thus in the Modules directory of the MarkLogic installation, not in the modules filesystem location or modules database of the AppServer.
  ** The location is a filesystem location relative to MARKLOGIC_INSTALL_DIR/Modules on the MarkLogic server. This has nothing to do with the xdmp:modules-root().
• You can't fetch filesystem locations via xdmp:get-document-get(), I think you would need to access them somehow in the filesystem of the server via something like xdmp:filesystem-file(fn:concat("/opt/MarkLogic/Modules", "/MarkLogic/security.xqy"));, and to have the proper permissions for that! I don't know how to retrieve the path of Marklogic's Modules directory from within XQuery or how to evaluate the environment variable MARKLOGIC_INSTALL_DIR.
• The same is true for the file locations of Modules of an AppServer that stores modules in the filesystem: you have to use something like xdmp:filesystem-file, but now obeying xdmp:modules-root().
• Modules in a Modules database you can fetch wia xdmp:document-get(). Have a look at xdbc-selector that does it. It should work just by selecting the Modules database as content-base.
• Keep in mind to obey the rules on how to resolve modules paths as decribed in https://docs.marklogic.com/guide/app-dev/import_modules#id_29407 That is, in case you don't have absolute paths already.

My perspective on priorities:

• For now, we care just for Modules in a modules database.
• We don't put code in the filesystem.
• We also don't put custom code Modules in the MarkLogic installation directory (also, this is something MarkLogic strongly advises against).
• While it is really nice when error reporting also allows to access the source code of the server implementation and not only for the user code, I would regard this as additional convenience.

Cheers,
Max

[m-g-r]

PS: fn:concat(xdmp:modules-root(), "/MarkLogic/security.xqy") might lead to a double slash.

[proofit404]

As we say in Russia:

Either I'm a fool, or my skis not slide.

I've spent yet another two days trying to import a module from the modules database :(

I've tried every possible combination of xdmp:document-insert, xdmp:document-load, xdmp:permission. I just can't create importable module for my test purposes.

It will be very nice if you can write short instruction for dummies how to create importable module, how to upload it in to modules database, how to import it.

On progress

I introduced configurable variable for MarkLogic installation path. By default its value is "/opt/MarkLogic/". I need to note that Windows support will require slightly more efforts to implement. Now when we navigates through error list for each module uri we tries:

1. Find this module in /path/to/MarkLogic/Modules/uri/part/of/path.xqy This works reasonable well, if you setup this variable correctly.
2. If App Server uses (file system) as its modules storage, we search given module on file system. This technique was discussed earlier.
3. If App Server uses Modules database, we will get this module content using fn:doc.

I need to test last condition, by I failed to build necessary environment for it.

Regards, Artem.

[m-g-r]

Hi Artem,

Thanks that sounds quite. I'll make you a test example later. No worries, sometimes things behave strangely. Also I will give you feedback about the possible renaming etc.

Cheers Max

[m-g-r]

Hi,

here is a complete example.

A module:
foo-module.xqy.txt

XQuery code to upload to MarkLogic when talking directly to the Modules database:
foo-module-upload.xqy.txt

XQuery code to upload to MarkLogic when talking directly to another database database:
foo-module-upload-from-content-base.xqy.txt

XQuery code to show the module when talking directly to the Modules database:
foo-module-show.xqy.txt

XQuery code to show the module when talking to antother database:
foo-module-show-from-content-base.xqy.txt

In general: Better not use xdmp:document-get()``, as we never want to fetch files from the web (i.e., over http). Either usedocto fetch from a xml database or usexdmp:filesystem-file``` to retrieve files form the filesystem of the server.

Hope that helps,
Max

[proofit404]

I've followed your recommendations and it looks like it finally works. Thanks!!

Please check that it works in your environment too.

I still can't reproduce the case with empty error buffer. Maybe you will find those circumstances later.

NOTE: you should use error-buffer branch in the Oook repository since we've moved the code and not the issues.

[m-g-r]

Thank you! That sounds great. Currently, I'm very busy with a project and cannot change the tooling right now. But I noticed your code changes and I'm eager to use it. Thanks for already making the oook-selector repository and updating the code!

Oh, could you post screenshot of the error-buffer output? Did you rearrange the output a bit as suggested at the end of my comment in #13 (comment)?

By the way, for said project I use all these facilities. Nice to use, also when talking to several MarkLogic instances at the same time. And also some queries are not only more convenient to do from Emacs than MarkLogic's Query Console but also some huge queries I get in below a second of response time while the Query Console takes 4-7 seconds. Isn't that nice? (I'm guessing that the Javascript does some post-processing for nice presentation that takes longer.)

Cheers,
Max

[proofit404]

Oh, could you post screenshot of the error-buffer output?

Did you rearrange the output a bit as suggested at the end of my comment in #13 (comment)?

Not yet, but I found the way which can probably work out. If this research will be successful we will have new output format soon.

By the way I'm glad you're like it!

[proofit404]

Good morning!

I rearrange the output as suggested at the end of your comment.

Basically I reimplement exception representation from scratch in the clojure handler. It supersedes this and that methods.

Please test if it works for you and if it fits necessary information layout.

P.S. my clojure skills are weak so catch code looks not so good.

[m-g-r]

Very nice! I just have started to use it, so didn't have an in-depth look. So far very nice! Is there a key to show also the full Java stack trace? Just in case... Cheers, Max

[proofit404]

Yes, it's possible. I will add it at the end of buffer.

[proofit404]

Java stack trace now looks like this

There is a line at the end of the error buffer. It magically appended to original message somewhere outside of oook exception handler. I suspect this happens because we throw new exception within try/catch block. It mostly a cosmetic trouble so I don't research it too much.