Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problems with cx:eval #82

Closed
bertfrees opened this issue Feb 21, 2013 · 11 comments
Closed

Problems with cx:eval #82

bertfrees opened this issue Feb 21, 2013 · 11 comments

Comments

@bertfrees
Copy link

@bertfrees bertfrees commented Feb 21, 2013

cx:eval doesn't always work correctly. For example, when I replace this pipeline

<p:declare-step type="px:create-fileset" xmlns:p="http://www.w3.org/ns/xproc" xmlns:px="http://www.daisy.org/ns/pipeline/xproc" version="1.0">
    <p:output port="result" primary="true"/>
    <p:import href="fileset-create.xpl"/>
    <px:fileset-create base="file:/tmp"/>
</p:declare-step>

by this equivalent pipeline:

<p:declare-step type="px:create-fileset" xmlns:p="http://www.w3.org/ns/xproc" xmlns:px="http://www.daisy.org/ns/pipeline/xproc" xmlns:cx="http://xmlcalabash.com/ns/extensions" version="1.0">
    <p:output port="result" primary="true"/>
    <p:import href="http://xmlcalabash.com/extension/steps/library-1.0.xpl"/>
    <cx:eval>
        <p:input port="source">
            <p:empty/>
        </p:input>
        <p:input port="pipeline">
            <p:document href="fileset-create.xpl"/>
        </p:input>
        <p:input port="options">
            <p:inline>
                <cx:options>
                    <cx:option name="base" value="file:/tmp"/>
                </cx:options>
            </p:inline>
        </p:input>
    </cx:eval>
</p:declare-step>

I get a XD0023 error. I suspect the problem is that the function p:value-available() is not recognized inside cx:eval. I'm using Calabash 1.0.8-94.

(To try these examples you need some additional files: fileset-create.xpl and uri-functions.xsl)

@ndw
Copy link
Owner

@ndw ndw commented Mar 8, 2013

This is UGLY.

Extension functions need to be able to find their XProcRuntime object. So they squirrel it away in a ThreadLocal variable. But they're registered with the Saxon Processor object. So when the eval'd pipeline calls an extension function, it gets the wrong context.

I'm going to have to find some new solution. I talked to Mike about it briefly in Prague so I have an idea...

ndw added a commit that referenced this issue Mar 8, 2013
…val register extension functions with the correct context
@ndw
Copy link
Owner

@ndw ndw commented Mar 8, 2013

Can you try the latest saxon94 branch and see if it works for you?

I think it's fixed, but I get a different error running your example:

SEVERE: file:/Volumes/Data/github/calabash/fileset-create.xpl:err:XTSE0165:I/O error reported by XML parser processing http://www.daisy.org/pipeline/modules/file-utils/xslt/uri-functions.xsl: http://www.daisy.org/pipeline/modules/file-utils/xslt/uri-functions.xsl
SEVERE: Failed to compile stylesheet. 1 error detected.
SEVERE: Failed to compile stylesheet. 1 error detected.
SEVERE: Pipeline failed: net.sf.saxon.s9api.SaxonApiException: Failed to compile stylesheet. 1 error detected.
SEVERE: Underlying exception: javax.xml.transform.TransformerConfigurationException: Failed to compile stylesheet. 1 error detected.

However, I can reproduce that error outside of XML Calabash, so I don't think it's my bug :-)

@bertfrees
Copy link
Author

@bertfrees bertfrees commented Mar 12, 2013

Hello Norman,

Thanks a lot for helping me out. You're right, there was an error in my stylesheet, sorry for that.

I can confirm that your latest commit fixed the problem :)

Will do some additional testing...

Thanks,
Bert

@bertfrees
Copy link
Author

@bertfrees bertfrees commented Mar 14, 2013

I did some further testing and I couldn't find any more problems.

@ndw
Copy link
Owner

@ndw ndw commented Mar 15, 2013

Perfect. I'm going to call this one closed. If you run into more troubles, well, you know what to do :-)

@ndw ndw closed this Mar 15, 2013
@bertfrees
Copy link
Author

@bertfrees bertfrees commented Mar 15, 2013

Sure :)

@bertfrees
Copy link
Author

@bertfrees bertfrees commented Mar 18, 2013

Commit 9852719 must have broken something. When I run a script that does a lot of cx:eval, it becomes very slow and after a while I get a scary OutOfMemoryError:

org.eclipse.jetty.io.EofException: null
    at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:952) ~[na:na]
    at org.eclipse.jetty.http.HttpGenerator.complete(HttpGenerator.java:836) ~[na:na]
    at org.eclipse.jetty.server.AbstractHttpConnection.completeResponse(AbstractHttpConnection.java:603) ~[na:na]
    at org.eclipse.jetty.server.Response.complete(Response.java:1185) ~[na:na]
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:530) ~[na:na]
    at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:884) ~[na:na]
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938) ~[na:na]
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630) ~[na:na]
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) ~[na:na]
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77) ~[na:na]
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:622) ~[na:na]
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46) ~[na:na]
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603) ~[na:na]
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538) ~[na:na]
    at java.lang.Thread.run(Thread.java:679) ~[na:1.6.0_27]
Caused by: java.io.IOException: Broken pipe
    at sun.nio.ch.FileDispatcher.write0(Native Method) ~[na:1.6.0_27]
    at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.6.0_27]
    at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:122) ~[na:1.6.0_27]
    at sun.nio.ch.IOUtil.write(IOUtil.java:78) ~[na:1.6.0_27]
    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:352) ~[na:1.6.0_27]
    at org.eclipse.jetty.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:288) ~[na:na]
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:356) ~[na:na]
    at org.eclipse.jetty.http.HttpGenerator.flushBuffer(HttpGenerator.java:888) ~[na:na]
    ... 14 common frames omitted
Exception in thread "Thread-19" java.lang.OutOfMemoryError: GC overhead limit exceeded
    at net.sf.saxon.tree.tiny.TinyTree.getNode(TinyTree.java:902)
    at net.sf.saxon.tree.tiny.PrecedingSiblingEnumeration.next(PrecedingSiblingEnumeration.java:45)
    at net.sf.saxon.tree.tiny.PrecedingSiblingEnumeration.next(PrecedingSiblingEnumeration.java:14)
    at net.sf.saxon.s9api.XdmSequenceIterator.hasNext(XdmSequenceIterator.java:51)
    at com.xmlcalabash.model.Step.recursiveAnonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.anonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.recursiveAnonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.anonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.recursiveAnonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.anonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.recursiveAnonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.anonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.recursiveAnonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.anonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.recursiveAnonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.anonymousName(Unknown Source)
    at com.xmlcalabash.model.Step.<init>(Unknown Source)
    at com.xmlcalabash.model.CompoundStep.<init>(Unknown Source)
    at com.xmlcalabash.model.DeclareStep.<init>(Unknown Source)
    at com.xmlcalabash.model.Choose.<init>(Unknown Source)
    at com.xmlcalabash.model.Parser.readChoose(Unknown Source)
    at com.xmlcalabash.model.Parser.readStep(Unknown Source)
    at com.xmlcalabash.model.Parser.readCatch(Unknown Source)
    at com.xmlcalabash.model.Parser.readStep(Unknown Source)
    at com.xmlcalabash.model.Parser.readTry(Unknown Source)
    at com.xmlcalabash.model.Parser.readStep(Unknown Source)
    at com.xmlcalabash.model.Parser.readOtherwise(Unknown Source)
    at com.xmlcalabash.model.Parser.readChoose(Unknown Source)
    at com.xmlcalabash.model.Parser.readStep(Unknown Source)
    at com.xmlcalabash.model.Parser.readForEach(Unknown Source)
    at com.xmlcalabash.model.Parser.readStep(Unknown Source)
    at com.xmlcalabash.model.Parser.parseDeclareStepBodyPassTwo(Unknown Source)
@ndw
Copy link
Owner

@ndw ndw commented Mar 19, 2013

Can you provide a pipeline that demonstrates this?

@ndw ndw reopened this Mar 19, 2013
@bertfrees
Copy link
Author

@bertfrees bertfrees commented Mar 20, 2013

I can't manage to reproduce the OutOfMemoryError with a simple example, but I can show you how the process becomes very slow. The time the conversion takes seems to increase exponentially with $high-number. When I set it to 2000, the process never returns.

<p:declare-step version="1.0" type="cx:eval_test"
            xmlns:p="http://www.w3.org/ns/xproc"
            xmlns:cx="http://xmlcalabash.com/ns/extensions"
            xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
            exclude-inline-prefixes="#all">
   <p:output port="result" primary="true"/>
   <p:import href="library-1.0.xpl"/>
   <p:xslt template-name="initial" name="xslt">
      <p:input port="source">
         <p:empty/>
      </p:input>
      <p:input port="stylesheet">
         <p:inline>
            <xsl:stylesheet version="2.0">
               <xsl:variable name="high-number" select="200"/>
               <xsl:template name="initial">
                  <a>
                     <xsl:for-each select="for $i in 1 to $high-number return $i">
                        <b/>
                     </xsl:for-each>
                  </a>
               </xsl:template>
            </xsl:stylesheet>
         </p:inline>
      </p:input>
      <p:input port="parameters">
         <p:empty/>
      </p:input>
   </p:xslt>
   <p:identity name="pipeline">
      <p:input port="source">
         <p:inline>
            <p:pipeline version="1.0">
               <p:identity>
                  <p:input port="source">
                     <p:inline><c/></p:inline>
                  </p:input>
               </p:identity>
            </p:pipeline>
         </p:inline>
      </p:input>
   </p:identity>
   <p:viewport match="b" name="viewport">
      <p:viewport-source>
         <p:pipe step="xslt" port="result"/>
      </p:viewport-source>
      <cx:eval>
         <p:input port="source">
            <p:pipe step="viewport" port="current"/>
         </p:input>
         <p:input port="pipeline">
            <p:pipe step="pipeline" port="result"/>
         </p:input>
         <p:input port="options">
            <p:empty/>
         </p:input>
      </cx:eval>
   </p:viewport>
</p:declare-step>
@ndw
Copy link
Owner

@ndw ndw commented Jul 30, 2013

I can reproduce the out-of-memory error for this, but I'm not sure I can fix it. This pipeline evaluates itself a fairly substantial number of times.

@bertfrees
Copy link
Author

@bertfrees bertfrees commented Jul 31, 2013

Yes you have a point. This issue is also not so critical anymore for us. Thanks for looking into it!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.