Designing ly.xml

[wbsoft]

The xml-export.ily script is now fairly complete, although we could change it in details or to improve usability. But now it can be tested and the XML it generates carefully examined. http://python-ly.readthedocs.org/en/latest/ly.xml.html#the-xml-export-ily-file.

Let's use this issue to discuss the XML output and see if we would need to change some aspects.

Then we design our own XML (built from the tokenized source, mimicking the source document) which will be clearly different:

• user variables are not substituted
• because we use it to write LilyPond code from scratch, it should be possible to add comments and newlines to the tree
• toplevel variables definitions and scheme expressions are to be preserved
• functions like \transpose and \relative are not executed already, so we need a way to specify LilyPond commands in their normal syntax.

It could be thinkable that we design two styles of XML tree:

1. very close to the source (like current ly.music but as XML, and akin to the XML xml-export.ily generates)
2. with all substitutions done and things like \relative and \transpose executed. This tree then is the same as the one the xml-export.ily script would generate and can be used to perform conversion to other formats and XML encodings.

The conversion from 1. to 2. should be very easy to implement. 1. is then used to understand a document as good as possible, but taylored to writing LilyPond sourcecode from scratch or altering the source document; and providing information (like how many notes are entered). 2. is used to export music to other formats. Also writing code from scratch from 2. could be implemented, but it would not much look like human-written LilyPond code.

[PeterBjuhr]

Great!

One question, do you mean that ly.music should hold functionality for conversions from 1 to 2? In some cases the conversion would be trivial but in others we would need to use xml-export.ily, wouldn't we?

[wbsoft]

I think that the conversion from 1 to 2 should be implemented in a ly.xml module. The ly.music probably remains untouched.

When LilyPond is available, exporting from LilyPond-generated output is the best way. But when it isn't or for simpler scores, we could create an XML tree ourselves.

[PeterBjuhr]

Agreed, we could get quite far without needing to use LilyPond and I guess that method would be quicker. (Could it even be up to the user to choose method when exporting?)

[PeterBjuhr]

I'm also wondering about the conversion in the other direction (from 2 to 1). Normally when a document is created from scratch we would like to go from 1 to a LilyPond source text, right (that format is closest to a document)? But if we want to import music (e. g. from MEI) would we then convert MEI -> 2 -> 1 -> LilyPond document? Or did you mean that it would be better to write LilyPond code from 2?

[wbsoft]

I think it should be designed so that creating LilyPond source from 2. would be perfectly possible. It'd just be that all music would be absolute and wrapped in one large expression inside a \score. An importer would probably create some assignments and build a document more like 1 maybe. So probably there's no strict distinction between 1 and 2...

[PeterBjuhr]

I was playing with the idea of using templates for imports (and perhaps also for creating documents from scratch). That way the user could have some control over what the imported document would look like.

I guess it's part of the idea that the template would be partly parsed into format 1 and then used to create the actual document.

Another additional use of such templates could be when creating a document via the score wizard.

[wbsoft]

Here's an example of the distance between the in-Lily-generated XML and what we would need:

\version "2.18.2"

\include "/home/wilbert/dev/python-ly/ly/xml/xml-export.ily"

\displayLilyXML {
  \clef treble
}

generates this scheme expression (when using \displayMusic):

(make-music
  'SequentialMusic
  'elements
  (list (make-music
          'ContextSpeccedMusic
          'context-type
          'Staff
          'element
          (make-music
            'SequentialMusic
            'elements
            (list (make-music
                    'PropertySet
                    'value
                    "clefs.G"
                    'symbol
                    'clefGlyph)
                  (make-music
                    'PropertySet
                    'value
                    -6
                    'symbol
                    'middleCClefPosition)
                  (make-music
                    'PropertySet
                    'value
                    -2
                    'symbol
                    'clefPosition)
                  (make-music
                    'PropertySet
                    'value
                    0
                    'symbol
                    'clefTransposition)
                  (make-music
                    'ApplyContext
                    'procedure
                    ly:set-middle-C!))))))

and generates this XML:

<?xml version="1.0" encoding="utf-8"?>
<music name="SequentialMusic">
  <origin filename="/home/wilbert/ly/lilyxml-test1.ly" line="5" char="16"/>
  <elements>
    <music name="ContextSpeccedMusic">
      <origin filename="/home/wilbert/ly/lilyxml-test1.ly" line="6" char="2"/>
      <element>
        <music name="SequentialMusic">
          <elements>
            <music name="PropertySet">
              <property name="value">
                <string value="clefs.G"/>
              </property>
              <property name="symbol">
                <symbol value="clefGlyph"/>
              </property>
            </music>
            <music name="PropertySet">
              <property name="value">
                <number value="-6"/>
              </property>
              <property name="symbol">
                <symbol value="middleCClefPosition"/>
              </property>
            </music>
            <music name="PropertySet">
              <property name="value">
                <number value="-2"/>
              </property>
              <property name="symbol">
                <symbol value="clefPosition"/>
              </property>
            </music>
            <music name="PropertySet">
              <property name="value">
                <number value="0"/>
              </property>
              <property name="symbol">
                <symbol value="clefTransposition"/>
              </property>
            </music>
            <music name="ApplyContext">
              <property name="procedure">
                <procedure name="ly:set-middle-C!"/>
              </property>
            </music>
          </elements>
        </music>
      </element>
      <property name="context-type">
        <symbol value="Staff"/>
      </property>
    </music>
  </elements>
</music>

When converting such XML we can easily recognize and understand the clef command that is used.
In the LilyPond script define-music-display-methods.scm we can see that the \displayLilyMusic command uses some logic to understand those property settings as a clef change.

But when we create an XML tree from source, we would like to have a simpler construction that also maps the clef change, e.g.:

<music name="_user_">
  <command name="\clef">
    <string value="treble"/>
  </command>
</music>

This is just a musing...

The same holds true for other commands that are already executed or parsed in some way by LilyPond when building the music expression, like \transpose, \relative, \override etc.

The idea is that we have a clear way to map things closely to LilyPond source. As \clef treble is clearly a music command, it should result in a <music> element, but after that it should be easily recognized as a piece of music that would need additional attention before it would look like the LilyPond-generated music. When building music from scratch, we would read and write such elements, but when handling in-LilyPond-generated XML we'll need to recognize the bunch of property-sets as a \clef command as well.

The <command> element is the special thing here, but we could use a different name or an XML namespace etc. We need to carefully design this :-)

I converted my Mouvement piece to XML, and the speed of loading it in xml.etree in Python3 and searching in it via XPath is fantastic!

[wbsoft]

I decided that for the base scheme types boolean, string, char, number and symbol I use the text value of the XML element as the value, not a value attribute. Other elements do not have text (only optional whitespace for indent).

So

\markup {
  \override #'(baseline-skip . 3) { Yo! }
}

now evaluates to:

<?xml version="1.0" encoding="utf-8"?>
<markup>
  <m name="line-markup">
    <m name="override-markup">
      <pair>
        <symbol>baseline-skip</symbol>
        <number>3</number>
      </pair>
      <string>Yo!</string>
    </m>
  </m>
</markup>

[wbsoft]

Well, I played with the LilyPond xml-export of the music structure and I propose the following approach:

1. we design an XML tree structure "format 1" for storing a tokenized source document (i.e. much like current ly.music)

2. the design of that XML closely follows the XML output generated by the xml-export.ily script. Especially:

   • Music resides in <music> elements, whereever possible we use the name attribute like LilyPond's
   • pitch and duration use the <pitch> and <duration> elements
   • we store basic values (i.e. string, number, etc) the same way
   • Markup is stored the same way
   • scheme expressions that are meant to be evaluated (like #(+ variable 3)) are stored in a <scheme> element
   • identifiers, assignments and \include statements are stored

3. we design an XML "format 2" where:

   • variables are expanded, also across included files
   • pitches in \relative music are put in the real octave, \transposed music is transposed
   • i.e. this music very much looks like the output of the xml-export.ily script

   Both XML variants should follow the same DTD and can be built manually. It is possible to convert from one format to another, also partly. Helper functions are written for that. The XML format is quite simple and uses only a few element types.

4. we make a module ly.xml.build with factory functions to make manually building the XML tree easier. The Frescobaldi Scorewizard will be able to use these.

5. we make a module ly.xml.event to iterate XML in a time-based way, much like current ly.music.event

6. we make a module ly.xml.ly2x that builds XML from tokenized source (much like current ly.music.read module). This module will probaly use ly.xml.build extensively.

7. we make a module ly.xml.x2ly which is concerned with producing LilyPond format output.

8. we make a module ly.xml.lilymusic which has functions like those in the define-music-display-methods.scm of LilyPond, that are capable of transforming complex music constructs back to the originating commands like \voiceOne, \override ... etc.

When exporting from a source ly file using python-ly, we will build a format 1 xml tree and (in most cases) convert that to a format 2 tree, which very much behaves like the xml-export.ily-generated XML. That tree then can be read and parsed into anything else, maybe even using XSLT. An events layer will be created to iterate in a time-based way, just like ly.music.event.

When it is desired, we can have LilyPond generate the XML tree, so music generated by LilyPond's scheme functions is available, using the xml-export.ily script. This is a powerful route, and the XML generated this way should also be fully understood by our script.

When importing music, it in most cases boils down to manually building the XML tree and then producing LilyPond source from it.

When the XML structure is done, we are able to understand music in a meaningful way and can implement a wealth of interesting functions like transpose, translate, change durations etc. etc.

When implementing such functions (that currently operate on the tokenized source), they should operate on the XML (format 1) tree and modify the tree via a layer that performs, but also stores the modifications. That layer then is able to perform the desired changes on the source document, without generating full parts again.

When all is done, ly.dom and ly.music are deprecated and removed.

[uliska]

I can't fully see through that but it looks very promising. (Unfortunately I had been away when all this discussion started, but i hope to be able to join efforts again slowly)

[PeterBjuhr]

I agree, it looks very good! But it means a massive rewrite of python-ly.

I have one question at this point: how is it decided when LilyPond and the xml-export.ily script is going to be used? I guess it may not be a Python-ly question; Python-ly should only provide the different possibilities!?? For the MEI export (as we have taken as example before) the question boils down to: should it be somehow decided from the source if xml-export.ily needs to be used or not, or should we just provide the different possibilities?

[PeterBjuhr]

we make a module ly.xml.build with factory functions to make manually building the XML tree easier. The Frescobaldi Scorewizard will be able to use these.

we make a module ly.xml.ly2x that builds XML from tokenized source (much like current ly.music.read module). This module will probaly use ly.xml.build extensively.

we make a module ly.xml.x2ly which is concerned with producing LilyPond format output.

Actually I have one more question: I'm a little confused by the description quoted above - isn't it ly.xml.x2ly that will be using ly.xml.build? It's in this direction we're building a new source...

[PeterBjuhr]

I agree, it looks very good! But it means a massive rewrite of python-ly.

I guess that means that we'll need to create a new branch where the current implementation stays while the restructure is in progress (like the v2.17.x branch in Frescobaldi). Or is it better to keep the master includable and develop in a branch?