New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problems with cx:eval #82

Closed
bertfrees opened this Issue Feb 21, 2013 · 11 comments

Comments

Projects
None yet
2 participants
@bertfrees

cx:eval doesn't always work correctly. For example, when I replace this pipeline

<p:declare-step type="px:create-fileset" xmlns:p="http://www.w3.org/ns/xproc" xmlns:px="http://www.daisy.org/ns/pipeline/xproc" version="1.0">
    <p:output port="result" primary="true"/>
    <p:import href="fileset-create.xpl"/>
    <px:fileset-create base="file:/tmp"/>
</p:declare-step>

by this equivalent pipeline:

<p:declare-step type="px:create-fileset" xmlns:p="http://www.w3.org/ns/xproc" xmlns:px="http://www.daisy.org/ns/pipeline/xproc" xmlns:cx="http://xmlcalabash.com/ns/extensions" version="1.0">
    <p:output port="result" primary="true"/>
    <p:import href="http://xmlcalabash.com/extension/steps/library-1.0.xpl"/>
    <cx:eval>
        <p:input port="source">
            <p:empty/>
        </p:input>
        <p:input port="pipeline">
            <p:document href="fileset-create.xpl"/>
        </p:input>
        <p:input port="options">
            <p:inline>
                <cx:options>
                    <cx:option name="base" value="file:/tmp"/>
                </cx:options>
            </p:inline>
        </p:input>
    </cx:eval>
</p:declare-step>

I get a XD0023 error. I suspect the problem is that the function p:value-available() is not recognized inside cx:eval. I'm using Calabash 1.0.8-94.

(To try these examples you need some additional files: fileset-create.xpl and uri-functions.xsl)

@ndw

This comment has been minimized.

Show comment
Hide comment
@ndw

ndw Mar 8, 2013

Owner

This is UGLY.

Extension functions need to be able to find their XProcRuntime object. So they squirrel it away in a ThreadLocal variable. But they're registered with the Saxon Processor object. So when the eval'd pipeline calls an extension function, it gets the wrong context.

I'm going to have to find some new solution. I talked to Mike about it briefly in Prague so I have an idea...

Owner

ndw commented Mar 8, 2013

This is UGLY.

Extension functions need to be able to find their XProcRuntime object. So they squirrel it away in a ThreadLocal variable. But they're registered with the Saxon Processor object. So when the eval'd pipeline calls an extension function, it gets the wrong context.

I'm going to have to find some new solution. I talked to Mike about it briefly in Prague so I have an idea...

ndw added a commit that referenced this issue Mar 8, 2013

Attempt to fix issue #82, make sure that 'inner pipelines' in an cx:e…
…val register extension functions with the correct context
@ndw

This comment has been minimized.

Show comment
Hide comment
@ndw

ndw Mar 8, 2013

Owner

Can you try the latest saxon94 branch and see if it works for you?

I think it's fixed, but I get a different error running your example:

SEVERE: file:/Volumes/Data/github/calabash/fileset-create.xpl:err:XTSE0165:I/O error reported by XML parser processing http://www.daisy.org/pipeline/modules/file-utils/xslt/uri-functions.xsl: http://www.daisy.org/pipeline/modules/file-utils/xslt/uri-functions.xsl
SEVERE: Failed to compile stylesheet. 1 error detected.
SEVERE: Failed to compile stylesheet. 1 error detected.
SEVERE: Pipeline failed: net.sf.saxon.s9api.SaxonApiException: Failed to compile stylesheet. 1 error detected.
SEVERE: Underlying exception: javax.xml.transform.TransformerConfigurationException: Failed to compile stylesheet. 1 error detected.

However, I can reproduce that error outside of XML Calabash, so I don't think it's my bug :-)

Owner

ndw commented Mar 8, 2013

Can you try the latest saxon94 branch and see if it works for you?

I think it's fixed, but I get a different error running your example:

SEVERE: file:/Volumes/Data/github/calabash/fileset-create.xpl:err:XTSE0165:I/O error reported by XML parser processing http://www.daisy.org/pipeline/modules/file-utils/xslt/uri-functions.xsl: http://www.daisy.org/pipeline/modules/file-utils/xslt/uri-functions.xsl
SEVERE: Failed to compile stylesheet. 1 error detected.
SEVERE: Failed to compile stylesheet. 1 error detected.
SEVERE: Pipeline failed: net.sf.saxon.s9api.SaxonApiException: Failed to compile stylesheet. 1 error detected.
SEVERE: Underlying exception: javax.xml.transform.TransformerConfigurationException: Failed to compile stylesheet. 1 error detected.

However, I can reproduce that error outside of XML Calabash, so I don't think it's my bug :-)

@bertfrees

This comment has been minimized.

Show comment
Hide comment
@bertfrees

bertfrees Mar 12, 2013

Hello Norman,

Thanks a lot for helping me out. You're right, there was an error in my stylesheet, sorry for that.

I can confirm that your latest commit fixed the problem :)

Will do some additional testing...

Thanks,
Bert

Hello Norman,

Thanks a lot for helping me out. You're right, there was an error in my stylesheet, sorry for that.

I can confirm that your latest commit fixed the problem :)

Will do some additional testing...

Thanks,
Bert

@bertfrees

This comment has been minimized.

Show comment
Hide comment
@bertfrees

bertfrees Mar 14, 2013

I did some further testing and I couldn't find any more problems.

I did some further testing and I couldn't find any more problems.

@ndw

This comment has been minimized.

Show comment
Hide comment
@ndw

ndw Mar 15, 2013

Owner

Perfect. I'm going to call this one closed. If you run into more troubles, well, you know what to do :-)

Owner

ndw commented Mar 15, 2013

Perfect. I'm going to call this one closed. If you run into more troubles, well, you know what to do :-)

@ndw ndw closed this Mar 15, 2013

@bertfrees

This comment has been minimized.

Show comment
Hide comment

Sure :)

@bertfrees

This comment has been minimized.

Show comment
Hide comment
@bertfrees

bertfrees Mar 18, 2013

Commit 9852719 must have broken something. When I run a script that does a lot of cx:eval, it becomes very slow and after a while I get a scary OutOfMemoryError:

org.eclipse.jetty.io.EofException: null
    at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:952) ~[na:na]
    at org.eclipse.jetty.http.HttpGenerator.complete(HttpGenerator.java:836) ~[na:na]
    at org.eclipse.jetty.server.AbstractHttpConnection.completeResponse(AbstractHttpConnection.java:603) ~[na:na]
    at org.eclipse.jetty.server.Response.complete(Response.java:1185) ~[na:na]
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:530) ~[na:na]
    at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:884) ~[na:na]
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938) ~[na:na]
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630) ~[na:na]
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) ~[na:na]
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77) ~[na:na]
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:622) ~[na:na]
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46) ~[na:na]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603) ~[na:na]
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) ~[na:na]
    at java.lang.Thread.run(Thread.java:679) ~[na:1.6.0_27]
Caused by: java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcher.write0(Native Method) ~[na:1.6.0_27]
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.6.0_27]
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:122) ~[na:1.6.0_27]
    at sun.nio.ch.IOUtil.write(IOUtil.java:78) ~[na:1.6.0_27]
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:352) ~[na:1.6.0_27]
    at org.eclipse.jetty.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:288) ~[na:na]
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:356) ~[na:na]
    at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:888) ~[na:na]
    ... 14 common frames omitted
Exception in thread "Thread-19" java.lang.OutOfMemoryError: GC overhead limit exceeded
    at net.sf.saxon.tree.tiny.TinyTree.getNode(TinyTree.java:902)
    at net.sf.saxon.tree.tiny.PrecedingSiblingEnumeration.next(PrecedingSiblingEnumeration.java:45)
    at net.sf.saxon.tree.tiny.PrecedingSiblingEnumeration.next(PrecedingSiblingEnumeration.java:14)
    at net.sf.saxon.s9api.XdmSequenceIterator.hasNext(XdmSequenceIterator.java:51)
    at com.xmlcalabash.model.Step.recursiveAnonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.anonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.recursiveAnonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.anonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.recursiveAnonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.anonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.recursiveAnonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.anonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.recursiveAnonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.anonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.recursiveAnonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.anonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.<init>(Unknown Source)
    at com.xmlcalabash.model.CompoundStep.<init>(Unknown Source)
    at com.xmlcalabash.model.DeclareStep.<init>(Unknown Source)
    at com.xmlcalabash.model.Choose.<init>(Unknown Source)
    at com.xmlcalabash.model.Parser.readChoose(Unknown Source)
    at com.xmlcalabash.model.Parser.readStep(Unknown Source)
    at com.xmlcalabash.model.Parser.readCatch(Unknown Source)
    at com.xmlcalabash.model.Parser.readStep(Unknown Source)
    at com.xmlcalabash.model.Parser.readTry(Unknown Source)
    at com.xmlcalabash.model.Parser.readStep(Unknown Source)
    at com.xmlcalabash.model.Parser.readOtherwise(Unknown Source)
    at com.xmlcalabash.model.Parser.readChoose(Unknown Source)
    at com.xmlcalabash.model.Parser.readStep(Unknown Source)
    at com.xmlcalabash.model.Parser.readForEach(Unknown Source)
    at com.xmlcalabash.model.Parser.readStep(Unknown Source)
    at com.xmlcalabash.model.Parser.parseDeclareStepBodyPassTwo(Unknown Source)

Commit 9852719 must have broken something. When I run a script that does a lot of cx:eval, it becomes very slow and after a while I get a scary OutOfMemoryError:

org.eclipse.jetty.io.EofException: null
    at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:952) ~[na:na]
    at org.eclipse.jetty.http.HttpGenerator.complete(HttpGenerator.java:836) ~[na:na]
    at org.eclipse.jetty.server.AbstractHttpConnection.completeResponse(AbstractHttpConnection.java:603) ~[na:na]
    at org.eclipse.jetty.server.Response.complete(Response.java:1185) ~[na:na]
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:530) ~[na:na]
    at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:884) ~[na:na]
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938) ~[na:na]
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630) ~[na:na]
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) ~[na:na]
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77) ~[na:na]
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:622) ~[na:na]
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46) ~[na:na]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603) ~[na:na]
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) ~[na:na]
    at java.lang.Thread.run(Thread.java:679) ~[na:1.6.0_27]
Caused by: java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcher.write0(Native Method) ~[na:1.6.0_27]
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.6.0_27]
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:122) ~[na:1.6.0_27]
    at sun.nio.ch.IOUtil.write(IOUtil.java:78) ~[na:1.6.0_27]
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:352) ~[na:1.6.0_27]
    at org.eclipse.jetty.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:288) ~[na:na]
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:356) ~[na:na]
    at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:888) ~[na:na]
    ... 14 common frames omitted
Exception in thread "Thread-19" java.lang.OutOfMemoryError: GC overhead limit exceeded
    at net.sf.saxon.tree.tiny.TinyTree.getNode(TinyTree.java:902)
    at net.sf.saxon.tree.tiny.PrecedingSiblingEnumeration.next(PrecedingSiblingEnumeration.java:45)
    at net.sf.saxon.tree.tiny.PrecedingSiblingEnumeration.next(PrecedingSiblingEnumeration.java:14)
    at net.sf.saxon.s9api.XdmSequenceIterator.hasNext(XdmSequenceIterator.java:51)
    at com.xmlcalabash.model.Step.recursiveAnonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.anonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.recursiveAnonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.anonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.recursiveAnonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.anonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.recursiveAnonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.anonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.recursiveAnonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.anonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.recursiveAnonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.anonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.<init>(Unknown Source)
    at com.xmlcalabash.model.CompoundStep.<init>(Unknown Source)
    at com.xmlcalabash.model.DeclareStep.<init>(Unknown Source)
    at com.xmlcalabash.model.Choose.<init>(Unknown Source)
    at com.xmlcalabash.model.Parser.readChoose(Unknown Source)
    at com.xmlcalabash.model.Parser.readStep(Unknown Source)
    at com.xmlcalabash.model.Parser.readCatch(Unknown Source)
    at com.xmlcalabash.model.Parser.readStep(Unknown Source)
    at com.xmlcalabash.model.Parser.readTry(Unknown Source)
    at com.xmlcalabash.model.Parser.readStep(Unknown Source)
    at com.xmlcalabash.model.Parser.readOtherwise(Unknown Source)
    at com.xmlcalabash.model.Parser.readChoose(Unknown Source)
    at com.xmlcalabash.model.Parser.readStep(Unknown Source)
    at com.xmlcalabash.model.Parser.readForEach(Unknown Source)
    at com.xmlcalabash.model.Parser.readStep(Unknown Source)
    at com.xmlcalabash.model.Parser.parseDeclareStepBodyPassTwo(Unknown Source)
@ndw

This comment has been minimized.

Show comment
Hide comment
@ndw

ndw Mar 19, 2013

Owner

Can you provide a pipeline that demonstrates this?

Owner

ndw commented Mar 19, 2013

Can you provide a pipeline that demonstrates this?

@ndw ndw reopened this Mar 19, 2013

@bertfrees

This comment has been minimized.

Show comment
Hide comment
@bertfrees

bertfrees Mar 20, 2013

I can't manage to reproduce the OutOfMemoryError with a simple example, but I can show you how the process becomes very slow. The time the conversion takes seems to increase exponentially with $high-number. When I set it to 2000, the process never returns.

<p:declare-step version="1.0" type="cx:eval_test"
            xmlns:p="http://www.w3.org/ns/xproc"
            xmlns:cx="http://xmlcalabash.com/ns/extensions"
            xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
            exclude-inline-prefixes="#all">
   <p:output port="result" primary="true"/>
   <p:import href="library-1.0.xpl"/>
   <p:xslt template-name="initial" name="xslt">
      <p:input port="source">
         <p:empty/>
      </p:input>
      <p:input port="stylesheet">
         <p:inline>
            <xsl:stylesheet version="2.0">
               <xsl:variable name="high-number" select="200"/>
               <xsl:template name="initial">
                  <a>
                     <xsl:for-each select="for $i in 1 to $high-number return $i">
                        <b/>
                     </xsl:for-each>
                  </a>
               </xsl:template>
            </xsl:stylesheet>
         </p:inline>
      </p:input>
      <p:input port="parameters">
         <p:empty/>
      </p:input>
   </p:xslt>
   <p:identity name="pipeline">
      <p:input port="source">
         <p:inline>
            <p:pipeline version="1.0">
               <p:identity>
                  <p:input port="source">
                     <p:inline><c/></p:inline>
                  </p:input>
               </p:identity>
            </p:pipeline>
         </p:inline>
      </p:input>
   </p:identity>
   <p:viewport match="b" name="viewport">
      <p:viewport-source>
         <p:pipe step="xslt" port="result"/>
      </p:viewport-source>
      <cx:eval>
         <p:input port="source">
            <p:pipe step="viewport" port="current"/>
         </p:input>
         <p:input port="pipeline">
            <p:pipe step="pipeline" port="result"/>
         </p:input>
         <p:input port="options">
            <p:empty/>
         </p:input>
      </cx:eval>
   </p:viewport>
</p:declare-step>

I can't manage to reproduce the OutOfMemoryError with a simple example, but I can show you how the process becomes very slow. The time the conversion takes seems to increase exponentially with $high-number. When I set it to 2000, the process never returns.

<p:declare-step version="1.0" type="cx:eval_test"
            xmlns:p="http://www.w3.org/ns/xproc"
            xmlns:cx="http://xmlcalabash.com/ns/extensions"
            xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
            exclude-inline-prefixes="#all">
   <p:output port="result" primary="true"/>
   <p:import href="library-1.0.xpl"/>
   <p:xslt template-name="initial" name="xslt">
      <p:input port="source">
         <p:empty/>
      </p:input>
      <p:input port="stylesheet">
         <p:inline>
            <xsl:stylesheet version="2.0">
               <xsl:variable name="high-number" select="200"/>
               <xsl:template name="initial">
                  <a>
                     <xsl:for-each select="for $i in 1 to $high-number return $i">
                        <b/>
                     </xsl:for-each>
                  </a>
               </xsl:template>
            </xsl:stylesheet>
         </p:inline>
      </p:input>
      <p:input port="parameters">
         <p:empty/>
      </p:input>
   </p:xslt>
   <p:identity name="pipeline">
      <p:input port="source">
         <p:inline>
            <p:pipeline version="1.0">
               <p:identity>
                  <p:input port="source">
                     <p:inline><c/></p:inline>
                  </p:input>
               </p:identity>
            </p:pipeline>
         </p:inline>
      </p:input>
   </p:identity>
   <p:viewport match="b" name="viewport">
      <p:viewport-source>
         <p:pipe step="xslt" port="result"/>
      </p:viewport-source>
      <cx:eval>
         <p:input port="source">
            <p:pipe step="viewport" port="current"/>
         </p:input>
         <p:input port="pipeline">
            <p:pipe step="pipeline" port="result"/>
         </p:input>
         <p:input port="options">
            <p:empty/>
         </p:input>
      </cx:eval>
   </p:viewport>
</p:declare-step>
@ndw

This comment has been minimized.

Show comment
Hide comment
@ndw

ndw Jul 30, 2013

Owner

I can reproduce the out-of-memory error for this, but I'm not sure I can fix it. This pipeline evaluates itself a fairly substantial number of times.

Owner

ndw commented Jul 30, 2013

I can reproduce the out-of-memory error for this, but I'm not sure I can fix it. This pipeline evaluates itself a fairly substantial number of times.

@bertfrees

This comment has been minimized.

Show comment
Hide comment
@bertfrees

bertfrees Jul 31, 2013

Yes you have a point. This issue is also not so critical anymore for us. Thanks for looking into it!

Yes you have a point. This issue is also not so critical anymore for us. Thanks for looking into it!

@ndw ndw closed this Jul 31, 2013

@bertfrees bertfrees referenced this issue Mar 29, 2017

Open

Identify performance bottlenecks #71

2 of 3 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment