Lines mode issue

[schrepfler]

I'm trying to process xml documents in line mode (one line each). When trying this kind of input

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><fixture start_time="1738166400000" event_id="9999" timestamp="1737369994942"/>
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><fixture start_time="1738065600000" event_id="9999" timestamp="1737369994951"/>
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><fixture start_time="1738164600000" event_id="9999" timestamp="1737369994960"/>

cat example.xmllines | xmq --lines
-:2: parser error : XML declaration allowed only at the start of the document
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><fixture start_time="1738
     ^
-:2: parser error : Extra content at the end of the document
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><fixture start_time="1738

Tried also cat example.xmllines | xmq, xmq --lines example.xmllines, xmq example.xmllines but it's all the same. Ideally I'd like to be able to pipe the output of kafka-console-consumer into it.

[schrepfler]

Also tried removing the xml declarations or having just one but then xmq is complaining about additional content at the end of the document (in essence it recognises only the first xml document).

[weetmuts]

You are right, working on a fix!

[weetmuts]

Ok, it is fixed, I think, and available in edge: https://github.com/libxmq/xmq/releases/tag/edge
(or if you want to build from source :-) )

Storing your example input in example.xmll
(using xmll to signify xml lines, xmql for xmqlines and jsonl for jsonlines)

xmq can now read xmll,xmql,jsonl files as long as you supply --lines.

You can for example apply the example.xslq transform on each line:
xmq --lines example.xmll transform example.xslq

prints

diff = 796405058,9999
diff = 695605049,9999
diff = 794605040,9999

where example.xslq contains:

xsl:stylesheet(version   = 1.0
               xmlns:xsl = http://www.w3.org/1999/XSL/Transform)
{
    xsl:template(match = /)
    {
        xsl:variable(name   = start
                     select = /fixture/@start_time)
        xsl:variable(name   = stamp
                     select = /fixture/@timestamp)
        xsl:variable(name   = diff
                     select = '$start - $stamp')
        diff {
            xsl:value-of(select = $diff)
            ','
            xsl:value-of(select = /fixture/@event_id)
        }
    }
}

You probably already know that there are xml-documents that cannot be transformed into a single line, those are documents containing newlines inside comments. But that is probably not a problem for you.

What is your use case here? Just curious. :-)

[schrepfler]

Hi @weetmuts, thanks for the quick turnaround, looking forward to the official release (I'm using brew for the install/update)!
We have a kafka topic with xml feed raw messages so I wanted to pipe it through xmq in order to improve readability of the payload a bit, I hope that makes sense.

[weetmuts]

The lines should be in the official brew install now! Please test!

Nice use case. This is exactly the type of use case I was thinking about.