Client unable to upload large XML files (4.1.0) #1874
Comments
|
How in the client are you uploading the file?
…On Mon, 14 May 2018, 06:15 thammr, ***@***.***> wrote:
When I try to upload a large XML file (>512k) using the client, I see the
following error in the Java console:
Exception in thread "Thread-23" java.lang.NullPointerException
at
org.exist.xmldb.RemoteCollection.uploadAndStore(RemoteCollection.java:606)
at
org.exist.xmldb.RemoteCollection.storeResource(RemoteCollection.java:507)
at
org.exist.xmldb.RemoteCollection.storeResource(RemoteCollection.java:476)
at org.exist.client.InteractiveClient.store(InteractiveClient.java:1763)
at org.exist.client.InteractiveClient.parse(InteractiveClient.java:1673)
at org.exist.client.ClientFrame$2.run(ClientFrame.java:980)
I tried to reconstruct the code to understand what the problem is.
It appears the res.getContent() returns a String (not a File or
InputStream). Consequently the InputStream instance "is" is never
instantiated and generates a null pointer exception.
This bug also causes restore to skip large files.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#1874>, or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABNJuQc5FFLFQW-Ff8rsduGayEO4w0lbks5tyNOogaJpZM4T9BnS>
.
|
|
Selecting the Add Document button on the toolbar gives the above error if the file is large. The upload dialog simply hangs. |
|
Please fill in the Issue template as a lot of essential information is missing here, making this a potential long ping-pong thread with a lot of followup questions. For next ticket please use the template otherwise we'll not be able to pickup the issues reported effectively. |
|
Where do I find the Issue Template? |
|
You’ll see it whenever you create a new issue. It’s supplied from https://github.com/eXist-db/exist/blob/develop/.github/ISSUE_TEMPLATE.md. |
|
Problem Client cannot upload large XML files. If I upload such a file by clicking on the add resource button, the upload fails to complete. The java console shows the stack-dump, I initially reported. Expected I expect the client to upload large XML files without error. As far as I know, the client provides the only convenient way of creating and restoring partial backups. This is useful when upgrading eXist. Reproduce In my experience, any attempt to upload a large XML file using the client fails. Context eXist 4.1.0 |
|
Hmm I was trying to reproduce this, and found that on macOS 10.13.4 i cannot use the java web client at all see java bug. Somehow when trying to start the jnlp it cannot detect java. So not much help, i did restore a backup with some large file (10mb and up) without problems. |
|
We are also experiencing similar issue. When we are trying to upload large file using Java web client, it just continuous forever without any error. We are also seeing this error while doing the database restore then we get following error message, Exception: ---------------------------------------- We had very simple xml file with simple structure but with high repetition of nodes shown in below example. So the main issue is not because of large file (the above file fails with just less than 1 MB size) but with high repetition of xml nodes, like in the above example if |
|
@iampopuri what version of eXist-db ? I am pretty sure this has been fixed ( i had similar issues, like you describe). this ticket is about a different problem. |
|
4.1.0 |
|
Hi all, We have the same error with files larger than 512Mb. We have installed eXist-db 4.1 (project.built=201804161617 scm.revision=919514c68) on a server and our java application is writing some xml files in the database. To do it, we use the method "storeResource" in the class org.exist-db.xmldb. If the files length is greater than 512K, we got the same error as mentionned by @thammr above: Caused by: java.lang.NullPointerException It seems that the "res.getContent" within the uploadStore method returns a byte[] and not a file or an input stream. |
|
I can confirm the same issue :-( with the webstart client uploading a large file fails for v4.1 :-(
client side: (log) serverside: Code: |
|
👍 One of our clients just encountered the same problem using Oxygen 20.0 to upload files to an Exist 4.1 server. |
|
Yes, it was me. Also, the problem I'm having concerns any file size (for example an .xml file weighing 57KB). This is the error I get when I drag and drop a file to my eXist-db 4.1.0 collection from oXygen 20: |
dizzzz
added
bug
|
👍 And yet another Oxygen XML Editor + Exist 4.1 client encountered this problem. |
|
Hello, I'm not sure this is related to the same issue: if I edit an existing file in oXygen and hit 'save', I get: eXist-db 4.1.0, oXygen 20 |
|
All traces show an NPE with line 606 of RemoteCollection#uploadAndStore(), so yes, I expect so. |
|
@lguariento Possible you need extra JAR libraries loaded by the data source configured on the client side. |
|
@raducoravu That's what I always use. |
|
@raducoravu Oddly (at least for me), after closing and reopening oXygen the connection opens correctly, and I can, at last, save and upload* documents without issues now. Problem solved, as far as I am concerned! Thanks to all. *I spoke too soon (see below) :( |
|
👍 Great, Oxygen has some internal class loader caches and this is probably why reopening it solved the problem. |
|
|
|
Does oxygen have the latest eXist jars? |
|
@adamretter Is it somethign I should check? That's what I see from my end. |
|
@lguariento @adamretter I will try to install Exist 4.2 on my side and reproduce the hang, then add some logging and see where it hangs. |
|
Thanks @raducoravu |
|
We are uploading the resource on a separate thread (because we want to show the progress). I reproduced the problem and connected to Oxygen using jconsole, the thread is active (it's not blocked) and it seems to keep saving the resource over and over again. The thread stack trace at a certain moment in time looks something like this: Oxygen calls "RemoteCollection.storeResource" only once (I added some logging to make sure of this) but somehow the Exist code seems to repeat numerous times either the writing of the resource or the closing of the output stream (which triggers the HTTP PUT to be called in Oxygen's custom HTTP url connection implementation). |
|
Thanks @raducoravu . Since this has proven to be (still) a problem, can someone re-open the issue? Or shall we open a new issue? |
|
@dizzzz I think it was you that submitted a patch to fix this - if you have a moment, could you take a look? I won't be able to get to this just yet |
|
OK... |
|
reference: #1883 |
|
@dizzzz Looking over your commits, a fix like this: is in my opinion not good because it breaks the original encoding of the string. On Windows for example the chars will be converted to bytes using the Cp1252 encoding which is very restrictive. Ideally you could detect the encoding from the XML string by looking at the XML declaration PI, if you cannot detect it then fallback to "UTF8" and do something like ".getBytes("UTF8")" About the current problem with the XML content being continuously POST-ed by Oxygen to the Exist server, where can I find the latest code for the "RemoteCollection.java"? Is it this one: https://github.com/eXist-db/exist/blob/develop/src/org/exist/xmldb/RemoteCollection.java ? Because the Java code does not seem to fit with the line/column information I obtained on the stack trace I logged above (RemoteCollection.uploadAndStore(RemoteCollection.java:625)). and that particular FastByteArrayInputStream input stream implementation might start returning 0 when the end of stream is reached instead of -1 and remain in the is.read loop. |
|
Found the problem, if you log the "chunk.length" before the while, it's "0", so you will keep on reading 0 bytes from the stream and sending it to the server. or you may investigate further into why the "getStreamLength()" and "getContentLength()" may return "0". |
|
@dizzzz @raducoravu we are also getting below error due to following code. See the exception log below
Buildfile: C:\Localdata\eXist-db-4.2.0\samples\ant\build.xml restore: |
|
@adamretter As the fix is only client-side any chance for me and @lguariento to get our hands on an "exist.jar" with the modification incorporated? So that we can test and verify this fix with Oxygen before the next Exist release... |
|
@raducoravu The latest builds are always here - http://static.adamretter.org.uk/exist-nightly/ however these will be in the 5.0.0-SNAPSHOT line now. Not sure if that is what you want? We just released 4.2.0 and 5.0.0-RC1. There will also be a 4.2.1 shortly |
|
Thanks @adamretter, then we'll wait for the 4.2.1 release. |
|
@raducoravu I just installed eXist-db 4.2.1, recreated the oXygen connection, and I can now upload .xml files from local to the server via oXygen, and also save large files without getting weird errors. All solved for me! :) |
|
@lguariento Cool, thanks for updating the thread. |
|
thnx guys |
When I try to upload a large XML file (>512k) using the client, I see the following error in the Java console:
I tried to reconstruct the code to understand what the problem is.
It appears the res.getContent() returns a String (not a File or InputStream). Consequently the InputStream instance "is" is never instantiated and generates a null pointer exception.
This bug also causes restore to skip large files.
The text was updated successfully, but these errors were encountered: