Should global (system) notations be interspersed within regular parts?

[joeberkovitz]

Instead of placing within <measure> elements within <global>, should system notations be placed in individual parts and tagged in some way, perhaps via a scope="global" attribute or something similar?

This would avoid the problem of the <global> or similar element being very sparse with most measures containing nothing.

However, this approach could scatter system notations throughout the score in a happenstance way, since there is no particular part that makes sense for them to belong to.

While they could be forced to be placed in the first part, this leads to ambiguities as sometimes the first part is not visible on all systems. In MusicXML, the result is that system notations tend to appear in the topmost visible part, which varies unpredictably from score to score.

[joeberkovitz]

Chair discussion: this question also relates to part groups, which could share notations across their member parts. Would we encode this sharing also even though it's not score-wide? (e.g. Don Giovanni minuet).

[cschardt]

Why not have both?
Appearing in <system> is brief and convenient.
Appearing in the parts gives the opportunity to put individual displacements per part to each notation. How could this be achieved otherwise?

[jsutdolph]

I don't understand what the system element is. I have searched the document at https://w3c.github.io/mnx/specification/ and there is no system element described

[mscuthbert]

I have no problem with sparse usage especially if the encoding is optional. I think for musical analysis purposes it is much better to have everything that affects a measure of one part (whether it affects that measure alone or the whole system) to be either in the data for the part/measure itself or some other special purpose element and not in another part’s data hidden among information that only pertains to the other part.

[joeberkovitz]

Note that we changed the name of <system> to <global> in the initial spec draft -- see https://w3c.github.io/mnx/specification/#the-global-element

[fablau]

I second what @cschardt and @mscuthbert have proposed above, I don't see any other alternative at the moment. It is definitively convenient to have system notation references in separate parts. Unless we create a separate, "master" file which includes just system notation to which parts can always reference to.

[jsutdolph]

The trouble with global as described in the spec. is that this would not be easily readable by eye. You would need to count all those empty measure elements to ascertain whether the key change is (say) in bar 232. It would be better just to include non-empty measure elements and give them a number.

[jsutdolph]

Apropos cschardt 'why not have both' . I am very much against including redundant copies of the same information in a file. What does the client program do if the copies of the information are not in agreement? Is it forced to check this and handle it and make a decision? Do we have to define which is correct? An acceptable alternative would be to include the copied information in a comment.
What about having a single global element for all parts at the very start of the cwmn section.

[cecilios]

To answer this question, we need to clarify and agree on the purpose of the global element.

1. If the global element represents just common content (i.e. to facilitate not having to repeat things on every part) then it should not be mandatory to place common content in the global element. But then this common content must be repeated in each part.

2. But if it represents global constrains then only things that can not be changed in each part should be placed there. Common content not representing a constrain should not be allowed to be placed in global.

   If the global element represents constrains my answer to 'why not have both'? is: The idea of allowing global notations to be placed in individual parts and tagged in some way, perhaps via a scope="system" attribute would be redundant with the idea of having the global element. Having both is having alternative ways of encoding the same fact. As one of the objectives of MNX was to give clear directions on how to encode things and to avoid alternatives, having both approaches should not be allowed.

   The problem of the global element representing global constrains is that the common barlines must be always included (as is currently proposed, although we can change the decision), as this will force to include a lot of empty measures elements. It would be better just to include common barlines optional and require to include only measure elements containing global changes (time signatures, keys, etc.) and reference them by number or other reference. This could also facilitate dealing with multi-metric scores, as suggested in How are multi-metric scores are represented in CWMNX #55.

[joeberkovitz]

(Note: I'll switch from saying "system" to "global" to avoid further confusion over the name change.)

@cecilios: The intended purpose, which we could change, was for <global> to represent common content that can not be overridden by parts. The idea of encoding the same thing in two different ways (e.g. allowing global notations to occur within parts) goes against two of the main architectural goals of MNX: a) to eliminate alternative ways of doing the same thing, and b) to represent any particular piece of information once, in a single document location.

Note that I'm excluding cases like multi-metric scores, or scores with non-corresponding key signatures in different parts. (As you pointed out in #68, the very presence of <global> implies that certain classes of scores, e.g. multi-metric, cannot be represented -- and in these cases, one cannot meaningfully employ <global>.)

@cschardt @fablau: So, while it is undoubtedly convenient for some purposes to have global notations within parts, at the same time it's inconvenient in at least two ways that seem important:

• It creates many ways for documents to encode content with internal conflicts, which must all be checked.
• It requires a consumer implementation to scan all parts up front, in order to discover all global notations that may be scattered throughout them. This virtually guarantees a two-pass scan of the entire document.

In order to answer these objections, I think there needs to be a compelling case showing that embedding global notations in parts is more than just a convenience for a subset of applications.

@jsutdolph: I think you offer a good argument in favor of numbering.

[joeberkovitz]

I'll inject a new point here to consider, and I'm not sure yet how I feel about it. It relates to #55 as well as this issue.

It's been pointed out that <global> is not useful in the case of scores that are polymetric or polytonal. However, such scores often consist of one set of parts in one meter(key), and another set of parts in another meter(key). In these cases, would it make sense to be able to describe the constraints that apply to each subset of parts? To do this, we could generalize the concept of <global> to permit it to apply to any subset of parts, not only to all parts.

Since this would hugely expand the universe of possibilities that an application needed to deal with, I would expect it to be excluded from the standard profile. The question of the standard profile's definition is not resolved and is reflected in #68, by the way.

[mscuthbert]

Does the concept of <global> really become something like <part-group-settings> with a default part-group as applying to the entire score?

[cecilios]

@joeberkovitz

The intended purpose, which we could change, was for to represent common content that can not be overridden by parts.

I fully support this. This implies that it is not mandatory to include the common content. Therefore, including empty measure elements is optional. In this case, It should be considered not to allow to include empty measure elements as they do not add anything but noise. In this case, all measure elements included in global must be referenced by number or other.

would it make sense to be able to describe the constraints that apply to each subset of parts? To do this, we could generalize the concept of to permit it to apply to any subset of parts, not only to all parts.

My first impression is that this is a very interesting idea for dealing with polymetric and polytonal scores, that should be explored. Need to define a simple way for defining the subsets of parts. Perhaps overlapped sets should be allowed, for defining the content common to all parts, but this could create inconsistencies between them. Probably it would be much better to allow only disjoined sets at the expense of having to repeat the all parts common content in each global set.

this would hugely expand the universe of possibilities that an application needed to deal with

Not sure what are you referring to. An app. will first have to read the global element to get the common content and then use it for filling the gaps when reading each part. The only difference between having a global element or a few (probably not more than two or three) is splitting the global content in several tables indexed by part id. Not complex neither expensive.

[fablau]

@cschardt @fablau: So, while it is undoubtedly convenient for some purposes to have global notations within parts, at the same time it's inconvenient in at least two ways that seem important: 1. It creates many ways for documents to encode content with internal conflicts, which must all be checked. 2. It requires a consumer implementation to scan all parts up front, in order to discover all global notations that may be scattered throughout them. This virtually guarantees a two-pass scan of the entire document. In order to answer these objections, I think there needs to be a compelling case showing that embedding global notations in parts is more than just a convenience for a subset of applications.

@joeberkovitz I understand that, and of course that could be a problem. I am wondering then if there is a way to have something like a "global" or "master" file which can be used as reference for those global signs.

[bhamblok]

I think this is a very important discussion. Please allow me for a suggestion to think about for a couple of minutes. It's probably going to raise a new issue with a lot more questions, but... it might provide a solution for all of next issues:

• the current issue Should global (system) notations be interspersed within regular parts? #2 with global notation elements
• issue Distinguish directions that are not scoped to any particular sequence #32: "distinguish directions that are not scoped to any particular sequence"
• the polymetric issue How are multi-metric scores are represented in CWMNX #55 (Refine the definitions of MNX-Common profiles #68)
• "support for system flow and break control" issue Provide a way to encode system breaks and page breaks #57
• the "number of staves for part" issue [CWMNX] Question: Number of staves for a part #43
• a lot more issues to come

How does a musician/conductor read his music? He won't start to read part 1, finish it completely, then "backup" (~cf. musicXML) to the beginning of the piece to start to read part 2 to start over again for the next, and the next part...
Written music has evolved for hundreds of years and came to a form which is able to contain everything(?) we dreamed of. Why now change the order of information it contains?

Then again, I believe a (human) musician isn't reading in a strict "time-wise" sequence either, but rather in small chuncks: part-wise, yes, but rearly any further than a couple of beats.

So I'm questioning if the "part-wise" format is the most ideal solution! A more "human-approach" could work as well.

I would like to introduce the concept of a "sub-measure". Other element-naming suggestions could be "chunk" / "fraction" / "division" / please help me out here.

Here is a (very brief!) example-implementation:

<score>
  <cwmnx profile="standard">
    <measure-fraction number="1">
      <directions>... global directions ...</directions> (cf. issue #2)
      <part pid="1">
        <directions>... part directions ....</directions> (cf. issue #32)
        <sequence staff="1">...</sequence>
        <barline> (a polymetric barline cf. issue #55)
      </part>
      <part pid="2">
        <directions>... part directions ...</directions> (cf. issue #32)
        <sequence staff="1">...</sequence>
        <sequence staff="2">
          <directions>... staff/voice/sequence directions ...</directions>
        </sequence>
      </part>
    </measure-fraction>
    <measure-fraction number="1.x">
      <part pid="1">
        <sequence staff="1">...</sequence>
      </part>
      <part pid="2">
        <directions>...</directions> (cf. issue #32)
        <sequence staff="1">...</sequence>
        <sequence staff="2">...</sequence>
      </part>
      <barline> (a full-measure barline cf. issue #55)
    </measure-fraction>
    <measure-fraction number="2" system="new"> (cf. issue #57)
      <directions>... some new global directions ...</directions> (cf. issue #2)
      <part pid="1"> ... </part>
      <part pid="2"> ... </part>
      <part pid="x"> (cf. issue #43)
        <sequence staff="3">...</sequence> (cf. issue #43)
      </part>
    </measure-fraction>
  </cwmnx>
</score>

While I was writing this down, I think we simply can keep the element-naming for a "sub-measure" to simply be: "measure". Because in 99.x% of all cases, there won't be polymetry. But I want to keep the example above as is to point out a "measure" does not need to be a "full" measure.

This approach won't need a two-pass scan, but "mimics" the way we "as humans" read a paper piece of sheet music.
I would like to emphasize the importance of the hundreds of years of evolution in sheet music. Back then, people already realized music is time-wise (sort of speak), not part-wise. So, ..., why change the order of information now as it has been evolving for so many years?

[joeberkovitz]

@fablau: Please help me understand why the <global> element is not already exactly the same as the "master file" that you are proposing. While it is not a separate file, it contains exactly the information that you suggest, as far as I can tell, and there's no real need to have multiple files describing a score.

@bhamblok: For what it's worth, I have given a lot of thought to abandoning the partwise format (and this is not to dismiss any fresh thinking like yours). When all measures are in step (i.e. no polymetric music), this yields a reasonable structure with no need for fractions. In fact, MEI uses just such an approach. But with polymeter, it all falls apart. The use of measure fractions strikes me as giving up something very big in the name of a "human-like" principle that doesn't yield big benefits. The thing we give up is a direct and simple mapping between measures in the score and measures in the schema. In its place, developers would face a lot of complexity in fragmenting measures when writing CWMNX, and reassembling them when reading it, and music-reading humans are not the people who will be looking at an MNX file! My feeling to date has been that musical scores are a two-dimensional structure, of a kind that hierarchies are ill suited to represent. Neither the partwise nor timewise approaches are perfect, and we have to pick the least worst one :-)

[bhamblok]

@joeberkovitz : As you mention, when all measures are in step (i.e. no polymetric music), a timewise format yields a reasonable structure. Which could solve many more issues than we are creating while maintaining a partwise format.
However, when polymeter does occur, a whole lot of complexity (as a developer), regardless of the timewise format is added as such, while the direct and simple mapping between measures in the score, and measures in the schema is lost anyway. As well for the musician, as for the developer.

So I would still like to discuss the topic of a timewise format a bit further.
We could keep a measure schema in most music (non polymetric music). While in others, we could approach the schema like the image below. Each "measure" (-fraction) takes the number of beats, up until some part reaches a "breakpoint"/barline.

original picture taken from the Lilypond blog: http://lilypondblog.org/2014/05/independent-meters

[joeberkovitz]

After thinking about this a while longer, I began to ask myself whether there were other ways to look at this problem besides MusicXML's timewise or partwise options, or @bhamblok's suggestion of breaking measures up into little pieces. I think I have found at least one such alternative. While I'm not convinced it's the best solution yet, it's worth including in the discussion. It certainly has a different set of pros and cons, and perhaps has some appeal compared to the other avenues.

The basic approach is to identify subsets of parts that share an identical measure structure, and bundle these beneath a single element whose structure is somewhat "timewise". (For a non-polymetric score, there is only one such element, so the typical case is not complicated by this approach.) Within this element, the information we have been calling "global" is encoded once, and shared by all parts in the subset.

It's easiest to begin with an example of how this works for a non-polymetric score. Here, the new element <measures> encodes a stream of measures whose structure is shared by some set of parts. Each <measure> element now bundles up the former "global" information, plus one or more child elements describing that measure's content for each part. The <global> element thus disappears, and its contents migrate to the new hierarchy under <measure>.

<cwmnx profile="standard">
  <!-- set of measures for some parts that share a measure structure -->
  <measures>
    <!-- define all the parts that share this structure -->
    <part id="p1" style="y: above;">
      <part-name>Voice</part-name>
      <instrument-sound>voice.aa</instrument-sound>
    </part>
    <part id="p2" style="measure-numbering: none;">
      <part-name>Piano</part-name>
      <instrument-sound>keyboard.piano</instrument-sound>
    </part>

    <!-- Measure 1 in all parts -->
    <measure>
      <!-- first, directions common to this measure in all parts -->
      <directions>
        <tempo bpm="60" value="/4"/>
        <time signature="3/4"/>
        <key fifths="-3" mode="major"/>
        <instruction class="tempo-indication">Andantino</instruction>
      </directions>

      <!-- next, <part-measure> elements for each part with content in this measure -->
      <part-measure part="p1">
        <!-- directions for this measure applying only to the vocal part -->
        <directions>
          <staves number="1"/>
          <clef line="2" sign="G"/>
        </directions>
        <!-- content of this measure for the vocal part -->
        <sequence>
          <event measure="yes">
            <rest/>
          </event>
        </sequence>
      </part-measure>

      <part-measure part="p2">
        <!-- again, piano part-specific directions for this measure -->
        <directions>
          <staves number="2"/>
          <clef line="2" sign="G" staff="1"/>
          <clef line="4" sign="F" staff="2"/>
        </directions>
        <sequence staff="1">...</sequence>
        <sequence staff="2">...</sequence>
      </part-measure>
    </measure>

    <!-- Measure 2 in all parts -->
    <measure>
      <!-- (If there were directions common to both parts, they'd occur here) -->
      <part-measure part="p1">
        <directions>...</directions>
        <sequence>...</sequence>
      </part-measure>

      <part-measure part="p2">
        <directions>...</directions>
        <sequence staff="1">...</sequence>
        <sequence staff="2">...</sequence>
      </part-measure>
    </measure>
  </measures>
</cwmnx>

The approach to encoding a polymetric score is very straightforward: such a score includes multiple <measures> elements, one for each set of parts with the same metrical structure. In extreme polymetry where each part has a different structure, there would only be one <part> in each <measures>.

Pros:

• No need to correlate the <global> measure elements with the corresponding <part> measure elements. All of this information is now nested within a single <measure> element, and these occur sequentially.
• No empty <measure> elements; sparse encoding comes naturally to this structure. If there are no global directions in some measure, just leave them out. If a part has no music for some <measure>, then its <part-measure> can be omitted.

Cons:

• To parse the information for only a single part, consumers must process a filtered list of <part-measure> elements that only apply to that part.

[bhamblok]

@joeberkovitz: nice thinking, I like it :-) However, in polymetry, unfortunately there is another "con" in your proposal: where/how would we indicate system-brakes / page-flows?

[cecilios]

@joeberkovitz and @bhamblok : I like very musch your proposals.

The latest proposal from joeberkovitz does not solve the problem of encoding common directions for all parts (i.e. page breaks or system breaks) for polymetric music.

The approach suggested by bhamblok of using measure-fractions is much better to find a unified solution for all score types: normal scores (single time), polymetric and no-metric scores. The bias to preserving the measure element without any supporting use cases may be the root of current problems. By using measure-fractions:

• For normal music (a single common time signature) the measure-fraction is the measure (measure preserved for most music).

• For polymetric music, the measure-fraction could be something similar to what bhamblok proposed (I would suggest not splitting beamed groups and ties), but page break, system breaks and other notations common to all parts can be encoded only once for all parts.

• And for no-time music, measure-fractions are producer defined, and they will suggest feasible break points in all parts. But in any case page break and system breaks can also be encoded only once for all parts.

The measure-fraction elements provide feasible break points, so this simplifies line breaking algorithms for music not based on measures and could be and advantage for free flow layouts. Current algorithms must first identify suitable places for breaks an classify them by 'goodness' (i.e. for polymetric, better choose places with barline in all parts).

[joeberkovitz]

The co-chairs discussed this issue today at length. We recommend that the discussion focus on comparing the two alternatives that we feel are most viable:

1. Require the <measure> element to carry a number attribute which must sequentially number all measures in the score. This makes the correlation between measures in <global> and <part> elements clear, and allows empty measure elements to be omitted for sparse representation. At the same time, allow <global> elements to be provided for specific subsets of parts, thus supporting polymetric scores (perhaps outside the standard CWMN profile).

2. Reorganize both global measure content and part-based measure content as children of a single <measure> element as described here. This sacrifices placing all part-related content under a <part>, in order to to place all similarly-structured measure content under a <measure>.

In both of the above cases, directions that are shared by all staves of the music can be flagged as such via a specific attributes. This is only an issue for polymetric content, and we are not sure what sorts of directions would even be shared in such scores given the differences in measure layout, but it feels like more of an edge case than a central concern driving the design of the encoding.

We do not feel the following approaches are viable:

3. Including global content in part-specific measures. This is duplicative and violates the parsimonious-design principle of MNX.

4. Organizing polymetric scores in terms of measure fractions. Our objections include:

   1. This does not work for scores in which different parts progress at different tempos.
   2. Architecturally, it's bad to complicate an encoding without serious gains from doing so. System and line break positions could still be specified at any desired location without fragmenting measures, and common directions can be supported as described above.
   3. An event may be split across measure fractions, in which case it is not easy to tell from looking at the fraction what is going on inside it -- or, the representation of events gets subdivided, which is really overcomplicating things.
   4. Small, local changes in content (e.g. rebarring one part of a polymetric score) can force the entire score to be globally re-fragmented in a different way. This hinders use cases MC4: comparison and DEV3: dynamic score generation.

[mscuthbert]

Wonderful. Can I suggest that when numbering elements MNX distinguishes consistently between something like <measure sequence="12"> which means the 12th (or 13th if we decide to do 0-indexed sequence) and something like <measure number="12"> which means the measure that a musician would consider measure 12 (pickup measures, 1st/2nd endings, a score fragment beginning in media res etc. may interfere with this). It's possible that we might often see things like <measure sequence="11" number="234">

[mdgood]

In MusicXML 3.1 we use number for the identifier and text for the displayed measure number, but that is an artifact of the number attribute doing double duty in earlier MusicXML versions. Something like sequence and number sounds good to me.

[joeberkovitz]

@mscuthbert Yes, that's an excellent improvement (since this would be indeed a sequence number and not a musically-oriented number).

[mscuthbert]

Just to clarify -- is the "number attribute which must sequentially number all measures" to be called "sequence" not number, which would be reserved for the numbering as musicians use?

[joeberkovitz]

@mscuthbert my bad; I've edited the comment above to reflect the use of sequence as already agreed.

[joeberkovitz]

Note comments in #81: changed sequence to index to avoid confusion with the <sequence> element.

[joeberkovitz]

In the course of putting together PR #81 it became apparent that sparse encoding of measures could increase the effort for a consumer implementation to determine the actual makeup and length of the score. All parts must be decoded in full to discover the actual location of the last measure, since any part could always include <measure index="N"> where N is larger than any measure index seen so far.

Given this, do people have any second thoughts about sparse encoding? Placing back into Active Review to make sure.

[clnoel]

If this is truly a concern, I say we specify an optional(?) "max-measure-index" attribute as a courtesy to consuming implementations. For our current attempt to pin down the "standard" profile, we only need this attribute at the global level, since all parts are guaranteed to have the same number of measures, but eventually we'll need it at the part level to override for when we are allowing un-aligned measures.

I think that the advantage in reading and writing of being able to skip measures where nothing is happening overrides the concern about the max measure number.

[joeberkovitz]

I think the idea of providing a maximum is a very useful direction. I'd just prefer to see it required, rather than optional. This particular spec editor has a strong bias against courtesies -- having learned from experience that it leads straight back to the unpredictability that we're all trying to purge.

It could go right into <global>, as you say. (I'm not sure it would be needed elsewhere, even in the multi-metric case, because one has <global>s there too.

[clnoel]

OK, then required number. But, now that I'm thinking more, it should be measure count, not max index number (unless we are restricting the index numbers so that they start at 1 and never skip, so that some part has to reference each index number). And we should be specific as to whether "pick-up measures" count as a measure.

[joeberkovitz]

Good points. To clarify:

index is 1-based, so "count" and "maximum" are the same. And yes, "count" is the clearer term!

Pick-up measures absolutely do count in this sequence, because we're being clear that this has nothing to do with the user-visible numbering of measures (see PR #81 for the detailed language).

[cschardt]

Measures:

1. Measures have to be counted regardless of any musical meaning and numbering (pick-up, repeated numbering in first/second ending und whatnot) to keep clearity. They simply should count the xml-entities, shouldn't they?
2. Question: How should sequences with polymetric parts be handled? Assume a piece starting with
   3 measures in staff 1 against 2 measures in staff 2, after this continuing with common measures:

Staff 1:  |   1    |    2    |    3    |    4    |   5   | ...
Staff 2:  |      1      |       2      |    4    |   5   | ...

I never had to do with such compositions but I think, that this is the only resonable way of counting.

[mscuthbert]

Another possible way of counting could be

Staff 1:  |   1    |    3    |    5    |    6    |   7   | ...
Staff 2:  |      2      |       4      |    6    |   7   | ...

or even:

Staff 1:  |   1    |    2    |    3    |    4    |   5   | ...
Staff 2:  |      6      |       7      |    4    |   5   | ...

in either case, giving unique indexes for each measure item that has individual properties.

[joeberkovitz]

@cschardt That's exactly the proposed scheme in #81.

[joeberkovitz]

@mscuthbert These alternatives are feasible, but seem overcomplicated. The "namespace" for counting measures is only the set of global and part elements that share the same structure. Starting at 1 and counting up is easy and it works, since there cannot be collisions between measures in disjoint structures.

[mscuthbert]

@joeberkovitz gotcha -- does this mean that if there's one case of non-aligned measures in a part that the part must have its own separate set of <measure> elements for the rest of the piece?

[joeberkovitz]

@mscuthbert I think so... but an example would be good to make sure. There might be a case for allowing a part to elide some measures from its layout without changing the overall structure. If so then I would suggest it's a different issue.

[joeberkovitz]

Here's another alternative to throw out about the guess-the-score-length problem: require a <global> element to always encode the last measure in its content. If the last measure is empty, that forces the inclusion of an empty element <measure index="SCORE_LENGTH"/>.

[clnoel]

I could be mistaken, @joeberkovitz , but I believe that your newest suggestion might be harder to validate, and otherwise be functionally equivalent to adding a measure-count attribute.

[bhamblok]

You could call me tenacious, but ... :-)
This again wouldn't be of any issue if we choose for a more timewise structure (as discussed above).

[joeberkovitz]

Chairs: today we decided to remove the sparse-encoding feature from this issue as this facet of the change was generating too much controversy and needs further thought, so fixing it here was overstepping. Note that #52 already tracks that issue.