New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Backup-restore failure : parsing errors #1620

Closed
dizzzz opened this Issue Nov 19, 2017 · 18 comments

Comments

Projects
None yet
4 participants
@dizzzz
Member

dizzzz commented Nov 19, 2017

What is the problem

During a simple backup - restore (no database cleanup in between), some files haven't been restored due to errors reported by the XML parser:

Exception: ------------------------------------
Exception: ------------------------------------
Problems occured found during restore:

WARN: Failed to restore resource 'repo.xml'
from file '/path/eXist-backup.zip#db/apps/shared-resources/repo.xml'.
Reason: Failed to invoke method parse in class org.exist.xmlrpc.RpcConnection: org.xml.sax.SAXParseException: Element type "repo:meta" must be followed by either attribute specifications, ">" or "/>".

WARN: Failed to restore resource 'build.xml'
from file '/path/eXist-backup.zip#db/apps/webstart/build.xml'.
Reason: Failed to invoke method parse in class org.exist.xmlrpc.RpcConnection: org.xml.sax.SAXParseException: The string "--" is not permitted within comments.

WARN: Failed to restore resource 'repo.xml'
from file '/path/eXist-backup.zip#db/apps/monex/repo.xml'.
Reason: Failed to invoke method parse in class org.exist.xmlrpc.RpcConnection: org.xml.sax.SAXParseException: Element type "permissions" must be followed by either attribute specifications, ">" or "/>".

WARN: Failed to restore resource 'repo.xml'
from file '/path/eXist-backup.zip#db/apps/eXide/repo.xml'.
Reason: Failed to invoke method parse in class org.exist.xmlrpc.RpcConnection: org.xml.sax.SAXParseException: Element type "permissions" must be followed by either attribute specifications, ">" or "/>".

There is need a -- in comments (issue 2), and all of the repo.xml files contain a root element with an attribute which name starts with an # character:

<repo:meta xmlns="http://exist-db.org/xquery/repo" 
    xmlns:repo="http://exist-db.org/xquery/repo" 
   #unknown="http://exist-db.org/xquery/repo">
    <repo:description>Shared resources used by other apps</repo:description>
    <repo:author>Wolfgang Meier</repo:author>
    <repo:website>https://github.com/eXist-db/shared-resources</repo:website>
    <repo:status>alpha</repo:status>
    <repo:license>GNU-LGPL</repo:license>
    <repo:copyright>true</repo:copyright>
    <repo:type>library</repo:type>
    <repo:target>shared-resources</repo:target>
    <repo:prepare/>
    <repo:finish/>
    <repo:permissions user="admin" group="dba" mode="0775"/>

What did you expect

A restore should work flawlessly

Describe how to reproduce or add a test

In the admin client, make a backup of a database (no special apps needed) and directly do restore from the backup.zip file.

Context information

Please always add the following information

  • eXist-db version : latest dev
  • Java version (Java8u151)
  • Operating system (MacOs)
  • 64 bit
  • Any custom changes in e.g. conf.xml

@dizzzz dizzzz added the bug label Nov 19, 2017

@adamretter

This comment has been minimized.

Show comment
Hide comment
@adamretter

adamretter Nov 19, 2017

Member

The repo.xml file that you pasted above is not well-formed XML because of the #unknown="http://exist-db.org/xquery/repo".

Is the repo.xml actually present in the database? And if so, how did that happen? It should not be possible to store invalid XML.

Member

adamretter commented Nov 19, 2017

The repo.xml file that you pasted above is not well-formed XML because of the #unknown="http://exist-db.org/xquery/repo".

Is the repo.xml actually present in the database? And if so, how did that happen? It should not be possible to store invalid XML.

@dizzzz

This comment has been minimized.

Show comment
Hide comment
@dizzzz

dizzzz Nov 20, 2017

Member

Yes it is stored in the database....

Member

dizzzz commented Nov 20, 2017

Yes it is stored in the database....

@dizzzz

This comment has been minimized.

Show comment
Hide comment
@dizzzz

dizzzz Nov 20, 2017

Member

I’ll study further

Member

dizzzz commented Nov 20, 2017

I’ll study further

@pherk

This comment has been minimized.

Show comment
Hide comment
@pherk

pherk Nov 20, 2017

  • seems to be the same issue as reported in #1573 and #1587

pherk commented Nov 20, 2017

  • seems to be the same issue as reported in #1573 and #1587
@dizzzz

This comment has been minimized.

Show comment
Hide comment
@dizzzz

dizzzz Nov 20, 2017

Member

image

this is taken from a 'clean' database; even without backup/restore I see this #unknown attribute

Member

dizzzz commented Nov 20, 2017

image

this is taken from a 'clean' database; even without backup/restore I see this #unknown attribute

@dizzzz

This comment has been minimized.

Show comment
Hide comment
@dizzzz

dizzzz Nov 20, 2017

Member

@pherk indeed but I did not no a repair ; I'l mark this as a duplicate, thnx

Member

dizzzz commented Nov 20, 2017

@pherk indeed but I did not no a repair ; I'l mark this as a duplicate, thnx

@dizzzz

This comment has been minimized.

Show comment
Hide comment
@dizzzz

dizzzz Nov 20, 2017

Member

@adamretter it is actually a problem, we need to address this ; interesting that 'something' allows to write invalid XML

Note: there are 2 issues: the interesting "double dash" and the #unknown attribute

Member

dizzzz commented Nov 20, 2017

@adamretter it is actually a problem, we need to address this ; interesting that 'something' allows to write invalid XML

Note: there are 2 issues: the interesting "double dash" and the #unknown attribute

@dizzzz dizzzz added the duplicate label Nov 20, 2017

@adamretter

This comment has been minimized.

Show comment
Hide comment
@adamretter

adamretter Nov 20, 2017

Member

@dizzzz I have heard from @duncdrum and @wolfgangmm that your steps to reproduce this do not cause the issue for them. Can you tell me did you start from a clean database, or did you have an existing database?

I would love to jump into this, can you give me exact steps to reproduce from a clean install?

Member

adamretter commented Nov 20, 2017

@dizzzz I have heard from @duncdrum and @wolfgangmm that your steps to reproduce this do not cause the issue for them. Can you tell me did you start from a clean database, or did you have an existing database?

I would love to jump into this, can you give me exact steps to reproduce from a clean install?

@dizzzz

This comment has been minimized.

Show comment
Hide comment
@dizzzz

dizzzz Nov 21, 2017

Member

I’ll follow up, but I started with a clean database....

Member

dizzzz commented Nov 21, 2017

I’ll follow up, but I started with a clean database....

@dizzzz

This comment has been minimized.

Show comment
Hide comment
@dizzzz

dizzzz Nov 21, 2017

Member

Ok confirmed: how to reproduce:

\21 Nov 2017 19:03:57,579 [main] INFO  (JettyStart.java [run]:146) - [eXist Version : 3.6.0-SNAPSHOT] 
21 Nov 2017 19:03:57,579 [main] INFO  (JettyStart.java [run]:147) - [eXist Build : 201711211902] 
21 Nov 2017 19:03:57,579 [main] INFO  (JettyStart.java [run]:148) - [Git commit : 4b4a8a3fd] 
21 Nov 2017 19:03:57,579 [main] INFO  (JettyStart.java [run]:150) - [Operating System : Mac OS X 10.13.1 x86_64] 
21 Nov 2017 19:03:57,579 [main] INFO  (JettyStart.java [run]:151) - [log4j.configurationFile : file:///xxxxxxxx/exist/log4j2.xml] 
21 Nov 2017 19:03:57,583 [main] INFO  (JettyStart.java [run]:152) - [jetty Version: 9.4.6.v20170531] 
Member

dizzzz commented Nov 21, 2017

Ok confirmed: how to reproduce:

\21 Nov 2017 19:03:57,579 [main] INFO  (JettyStart.java [run]:146) - [eXist Version : 3.6.0-SNAPSHOT] 
21 Nov 2017 19:03:57,579 [main] INFO  (JettyStart.java [run]:147) - [eXist Build : 201711211902] 
21 Nov 2017 19:03:57,579 [main] INFO  (JettyStart.java [run]:148) - [Git commit : 4b4a8a3fd] 
21 Nov 2017 19:03:57,579 [main] INFO  (JettyStart.java [run]:150) - [Operating System : Mac OS X 10.13.1 x86_64] 
21 Nov 2017 19:03:57,579 [main] INFO  (JettyStart.java [run]:151) - [log4j.configurationFile : file:///xxxxxxxx/exist/log4j2.xml] 
21 Nov 2017 19:03:57,583 [main] INFO  (JettyStart.java [run]:152) - [jetty Version: 9.4.6.v20170531] 
@dizzzz

This comment has been minimized.

Show comment
Hide comment
@dizzzz

dizzzz Nov 21, 2017

Member

It looks like it is possible with the code that installs/deploys the JAR files, to install jar files that contain XML documents that are not invalid

  • the "#" attributes
  • the "--" double dash in an XML comment is not rejected
Member

dizzzz commented Nov 21, 2017

It looks like it is possible with the code that installs/deploys the JAR files, to install jar files that contain XML documents that are not invalid

  • the "#" attributes
  • the "--" double dash in an XML comment is not rejected
@adamretter

This comment has been minimized.

Show comment
Hide comment
@adamretter

adamretter Nov 28, 2017

Member

@dizzzz Okay so some good news. This is really really easy to reproduce. I have also written a test narrowing it down to an issue in the Memtree DOM (which I likely may have introduced during my DOM improvements) A fix will be coming soon...

Member

adamretter commented Nov 28, 2017

@dizzzz Okay so some good news. This is really really easy to reproduce. I have also written a test narrowing it down to an issue in the Memtree DOM (which I likely may have introduced during my DOM improvements) A fix will be coming soon...

@adamretter adamretter assigned adamretter and unassigned wolfgangmm Nov 28, 2017

@adamretter adamretter added this to the eXist-3.6.1 milestone Nov 28, 2017

@adamretter

This comment has been minimized.

Show comment
Hide comment
@adamretter

adamretter Nov 28, 2017

Member

@dizzzz Okay so there was an existing DOM bug in eXist-db prior to my DOM improvements (I am comparing against eXist-3.1.0) that meant that the repo.xml was not stored correctly, however the Exception would be swallowed in org.exist.repo.Deployment :-(

I have first fixed org.exist.repo.Deployment so that it doesn't swallow significant errors. I am now looking into fixing the underlying DOM issues (which don't seem entirely related to my DOM improvements).

screen shot 2017-11-28 at 22 07 58

  1. On the left of the screenshot is eXist-db 3.1.0 which throws an IllegalArgumentException in response to AttrImpl#getNodeName() returning xmlns:. This error is subsequently swallowed by bad exception handling code in org.exist.repo.Deployment.

  2. On the right hand side of the screenshot is eXist-3.6.0, which returns #unknown for AttrImpl#getNodeName() which is a valid QName, but not a valid attribute name, and so this is where the #unknown comes from.

Anyway long story short. There are several bugs, some pre-existing which were surfaced by improvements in the DOM, and some new as a result of improving the DOM.

Member

adamretter commented Nov 28, 2017

@dizzzz Okay so there was an existing DOM bug in eXist-db prior to my DOM improvements (I am comparing against eXist-3.1.0) that meant that the repo.xml was not stored correctly, however the Exception would be swallowed in org.exist.repo.Deployment :-(

I have first fixed org.exist.repo.Deployment so that it doesn't swallow significant errors. I am now looking into fixing the underlying DOM issues (which don't seem entirely related to my DOM improvements).

screen shot 2017-11-28 at 22 07 58

  1. On the left of the screenshot is eXist-db 3.1.0 which throws an IllegalArgumentException in response to AttrImpl#getNodeName() returning xmlns:. This error is subsequently swallowed by bad exception handling code in org.exist.repo.Deployment.

  2. On the right hand side of the screenshot is eXist-3.6.0, which returns #unknown for AttrImpl#getNodeName() which is a valid QName, but not a valid attribute name, and so this is where the #unknown comes from.

Anyway long story short. There are several bugs, some pre-existing which were surfaced by improvements in the DOM, and some new as a result of improving the DOM.

@adamretter

This comment has been minimized.

Show comment
Hide comment
@adamretter

adamretter Nov 29, 2017

Member

@dizzzz For the comments issue, it is reported in your log in /path/eXist-backup.zip#db/apps/webstart/build.xml. Where do I find a XAR that contains webstart/build.xml. Do I just build one from github.com/trundler/existdb-webstart?

Member

adamretter commented Nov 29, 2017

@dizzzz For the comments issue, it is reported in your log in /path/eXist-backup.zip#db/apps/webstart/build.xml. Where do I find a XAR that contains webstart/build.xml. Do I just build one from github.com/trundler/existdb-webstart?

@dizzzz

This comment has been minimized.

Show comment
Hide comment
@dizzzz

dizzzz Nov 29, 2017

Member

Hmmmm no I cleaned that one. I’ll have a look

Member

dizzzz commented Nov 29, 2017

Hmmmm no I cleaned that one. I’ll have a look

@dizzzz

This comment has been minimized.

Show comment
Hide comment
@dizzzz

dizzzz Nov 29, 2017

Member

i can reproduce it for this configuration:

29 Nov 2017 18:37:13,617 [main] INFO (JettyStart.java [run]:146) - [eXist Version : 3.6.0-SNAPSHOT]
29 Nov 2017 18:37:13,617 [main] INFO (JettyStart.java [run]:147) - [eXist Build : 201711291834]
29 Nov 2017 18:37:13,617 [main] INFO (JettyStart.java [run]:148) - [Git commit : 4b4a8a3]

test procedure:

  • create XAR file in eXide
  • download XAR file
  • unzip file
  • add XML file with content comments with double dash
  • build XAR file (run ant)
  • deploy XAR via package manager

the package manager is reporting the "--" issue, but the XAR file gets installed.

dashboard

the new XML file with -- is not present in the XAR file.
testtest-0.1.xar.renamedas.zip

This means that I cannot get a file containing a -- into a backup file....

Member

dizzzz commented Nov 29, 2017

i can reproduce it for this configuration:

29 Nov 2017 18:37:13,617 [main] INFO (JettyStart.java [run]:146) - [eXist Version : 3.6.0-SNAPSHOT]
29 Nov 2017 18:37:13,617 [main] INFO (JettyStart.java [run]:147) - [eXist Build : 201711291834]
29 Nov 2017 18:37:13,617 [main] INFO (JettyStart.java [run]:148) - [Git commit : 4b4a8a3]

test procedure:

  • create XAR file in eXide
  • download XAR file
  • unzip file
  • add XML file with content comments with double dash
  • build XAR file (run ant)
  • deploy XAR via package manager

the package manager is reporting the "--" issue, but the XAR file gets installed.

dashboard

the new XML file with -- is not present in the XAR file.
testtest-0.1.xar.renamedas.zip

This means that I cannot get a file containing a -- into a backup file....

@adamretter

This comment has been minimized.

Show comment
Hide comment
@adamretter

adamretter Nov 29, 2017

Member

@dizzzz Does this mean the issue is solved or not? Sorry I am not quite sure I understand your explanation.

Member

adamretter commented Nov 29, 2017

@dizzzz Does this mean the issue is solved or not? Sorry I am not quite sure I understand your explanation.

@dizzzz

This comment has been minimized.

Show comment
Hide comment
@dizzzz

dizzzz Nov 29, 2017

Member

For me it is solved as I cannot reproduce it (anymore).

It looks like the package installer correctly does not install faulty xml documents.

(Side note: should the package installer abort the xar install when a failure occurs?)

Member

dizzzz commented Nov 29, 2017

For me it is solved as I cannot reproduce it (anymore).

It looks like the package installer correctly does not install faulty xml documents.

(Side note: should the package installer abort the xar install when a failure occurs?)

@dizzzz dizzzz closed this in #1640 Dec 2, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment