Discussion: national rule layers can't be reached when BT-24 only declares the format (Factur-X / ZUGFeRD / bare EN16931)

[antoine-rehm]

First, thanks for phorm, phive-rules and ddd — the amount of ground they cover is remarkable, and I realise maintaining all of it is a lot. I'm opening this as a question rather than a bug report: I hit a concrete case with the French mandate, but while digging I started suspecting the underlying shape is not French-specific, and I'd rather check my understanding with you before proposing anything.

The concrete case

A Factur-X 1.0.9 (EXTENDED) CII invoice intended for the French mandate (Flux 2). Via /api/dd_and_validate, phorm reports no French national findings. Mustang 2.24.0 on the same file reports two:

BR-FR-09/BT-29 : Le SIRET doit contenir 14 chiffres et commencer par le SIREN.
BR-FR-10/BT-30 : Le SIREN du vendeur (ram:ID) est obligatoire et doit être composé exactement de 9 chiffres.

The rules themselves are fine in phive-rules: running its own 20260216_BR-FR-Flux2-Schematron-CII_V1.3.0.xslt on the document with Saxon gives byte-identical findings to Mustang. They are simply never selected.

Chain:

1. BT-24 = urn:cen.eu:en16931:2017#conformant#urn:factur-x.eu:1p0:extended
2. ddd value-providers.xml (syntax cii) → fr.factur-x:extended:latest
3. fr.factur-x:* is only an alias of de.zugferd:* (ZugferdValidation._createFacturXAlias(), ZUGFERD_TO_FACTURX_MAP); latest → de.zugferd:extended:2.5
4. That VES = FACTUR-X_EXTENDED.xsd + FACTUR-X_EXTENDED.xslt, nothing else. Zero occurrences of BR-FR / SIRET / SIREN; run standalone → 0 findings.
5. ddd reaches fr.ctc:extended-cii:latest only for BT-24 = urn:cen.eu:en16931:2017#conformant#urn.cpro.gouv.fr:1p0:extended-ctc-fr.

So fr.factur-x:extended:1.0.9 doesn't fail to apply BR-FR — it never contained them.

Why the obvious workaround doesn't apply

"Just emit the CTC-FR BT-24 instead" doesn't work: a Factur-X hybrid invoice must declare the Factur-X guideline URN in BT-24 for its profile, consistent with the PDF/A-3 XMP fx:ConformanceLevel. When the CII is extracted from the PDF and posted to dd_and_validate, BT-24 is necessarily the Factur-X URN.

Interestingly, neither schematron constrains the value of BT-24 — no occurrence of urn:factur-x.eu:1p0:extended in FACTUR-X_EXTENDED.xslt, and no occurrence of cpro.gouv.fr or factur-x.eu in the 1.4.0 CTC-FR / BR-FR-Flux2 sources. The national layer is orthogonal to the semantic profile; it composes cleanly on top. The obstacle is purely the routing table.

The part I think may generalise

BT-24 identifies a format/profile, not a jurisdiction. So whenever a country's rules are layered on top of a format-level profile, BT-24-only routing structurally cannot reach them. Looking at the cii syntax in value-providers.xml, the entries split cleanly in two:

• BT-24 declares the jurisdiction — de.xrechnung:cii:*. Unambiguous, works today.
• BT-24 declares only the format — de.zugferd:*, fr.factur-x:*, and eu.cen.en16931:cii:latest for bare urn:cen.eu:en16931:2017. No national layer is reachable, by construction.

France is where this bites right now, but the same shape shows up elsewhere:

• Germany. Mustang already does the equivalent: for ZUGFeRD/Factur-X BASIC and EN16931 profiles — whose BT-24 declares only the format — it additionally runs the German XRechnung CII schematron (XMLValidator.java:401-406), at notice severity by default (l.155), raised to error only when BT-24 really declares xrechnung. Same pattern, different country.
• Bare EN16931 and Peppol BIS. A French invoice using plain urn:cen.eu:en16931:2017, or Peppol BIS Billing 3.0, lands on eu.cen.en16931:cii:latest — also with no national layer.
• The French base profiles are entirely unmapped. ddd has no entry for fr.ctc:cii, fr.ctc:ubl-invoice or fr.ctc:ubl-creditnote — only the extended-* and cdar ones. Those base CIUS profiles are the mainstream mandate case, and they're unreachable for exactly the same reason: their BT-24 is shared with EN16931 / Peppol.

If that reading is right, then any future country layering national rules on Factur-X / ZUGFeRD / bare EN16931 CII will hit this identically — which is why I'd rather ask about the general shape than propose a France-only patch.

How Mustang resolves it

Not as an argument that Mustang is right, just as a data point on what a payload-based answer looks like. validator/src/main/java/org/mustangproject/validator/XMLValidator.java (2.24.0).

Discriminator — domestic FR transaction (l.226-230):

final String sellerCountry = getXPathString(doc, "…SellerTradeParty//PostalTradeAddress/CountryID/text()");
final String buyerCountry  = getXPathString(doc, "…BuyerTradeParty//PostalTradeAddress/CountryID/text()");
runFrenchCiiSchematron = "FR".equalsIgnoreCase(sellerCountry) && "FR".equalsIgnoreCase(buyerCountry);

That is BT-40 == FR and BT-55 == FR. Not BT-24, not BT-23 — no reference to BusinessProcess, ProcessID or cpro.gouv anywhere in the validator package.

Composition — additive, not a replacement (l.384-393):

if (xsltFilename != null) {
    validateSchematron(zfXML, xsltFilename, …, ESeverity.error);       // FACTUR-X_EXTENDED.xslt
    if (runFrenchCiiSchematron) {
        String xsltFRFilename = "/xslt/XP_Z12_012/20260216_BR-FR-Flux2-Schematron-CII_V1.3.0.xsl";
        validateSchematron(zfXML, xsltFRFilename, …, ESeverity.error);  // + BR-FR on top
    }
}

Mustang keeps the profile's Factur-X schematron as the primary validation and adds the national layer. This applies to every CII profile that set an xsltFilename.

Worth flagging: this means rerouting to fr.ctc:extended-cii:* would not be equivalent. That VES is EXTENDED-CTC-FR-CII.xslt + BR-FR-Flux2 — it replaces the Factur-X schematron with the French EN16931 derivative instead of layering on top. So "pick a different VESID" and "add a rule layer" are genuinely different operations here.

Possible directions

No strong opinion — this is where I'd value your view, especially on whether the general framing is the right one.

A — composite VES in phive-rules. Register e.g. Factur-X EXTENDED XSD + FACTUR-X_EXTENDED.xslt + BR-FR-Flux2-Schematron-CII as one VES (and equivalents per profile). Explicit, preserves Factur-X semantics, reproducible against Mustang. But it multiplies VES count per country × profile, and callers still need a way to select it.

B — phorm-side: let country drive the XML layer too. hybrid_validate?country=FR currently only feeds kaltblut's PDF-carrier layer (ApiPostHybridValidate, setCountry()); XML validation ignores it. Having country also select the national XML layer is self-contained, explicit, needs no payload heuristic, and generalises to any country. Would need exposing on dd_and_validate too.

C — payload heuristic in ddd. Two obstacles I ran into:

• ddd returns one VESID and has no notion of "add a layer", so C only helps combined with A.
• A data-only rule isn't expressible today: VPIfType is an xs:choice, so an <if> holds either set/flag or nested select, never both, and DDDValueProviderPerSyntax._getAllDeducedValuesRecursive() returns on the first <if> with determined values. "Default to fr.factur-x, override when FR/FR" can't be written; nesting a country select under the Factur-X <if> would leave every non-FR Factur-X document with no VESID at all. Two top-level selects relying on fall-through isn't deterministic either, since m_aSelects is a CommonsHashMap. Enabling C would need one of: set + nested select (schema 1.2, nested overriding outer); an explicit <else>; or an ordered m_aSelects with documented document-order evaluation.

D — document the limitation and leave selection to the caller. Reasonable, though dd_and_validate then silently under-validates, which is the surprising part.

Happy to do the work on whichever direction you think is right — including the ddd side — or to drop it if I've misread the design intent.

[phax]

@antoine-rehm thanks for the comprehensive discussion startpoint :-)

There are a couple of things I like to respond to, and where I would like to point out what others did:

BT-24

First of all, the BT-24 field "CustomizationID" (in EN 16931:2017) is defined as

An identification of the specification containing the total set of rules regarding semantic content, cardinalities and business rules to which the data contained in the instance document conforms.

and the following usage note:

This identifies compliance or conformance to this document. Compliant invoices specify: urn:cen.eu:en16931:2017. Invoices, compliant to a user specification may identify that user specification here. No identification scheme is to be used.

Additionally, EN 16931-1:2017 section 7.6 "Identification of CIUS" states:

each CIUS ... shall ... have an assigned identifier. The invoice instance document ... should carry the assigned identifier in ... 'Specification identification'. This will allow the receiver ... to apply processing ... in accordance with the rules under which it was generated.

Based on these definitions, my personal opinion on France's choice to use no specific BT-24 is, that I think they made a mistake.

Additionally, when looking at this from a German angle, where the PDF can be thrown away and only the XML can be processed, that seems to be a really bad design choice. To my understanding, the XML must be self-contained for processing - letting it depend on the PDF makes things even worse.

DDD

I am totally aware, that DDD is not the best possible solution, but it proves to be helpful for all specifications that follow the intended usage of BT-24 ;-)

Additionally, I am not a big fan of /dd_and_validate because it hands the "choice of rule selection" to the machine. For best validation results, imho the system users needs to know what rules to apply. Especailly in the scenario where BT-24 is incorrect errors like the one you are describing for France are inevitable. However /dd_and_validate seems to be the "most convenient" choice for users.

Country selection

Indeed using the country selection from the postal address seems to be a straight forward solution. However, e.g. in the Peppol BIS Billing Schematron rules, sometimes the countries are taken from the respective VAT number and sometimes even from the VAT number of the Tax Representative - it might be the case, that these cases don't apply to France, but they need to be kept in scope.

Ad Directions

composite VES in phive-rules

As you pointed out, it might be more correct to use the Factur-X Extended XSD instead of the "default" CII ones. Replacing the XSD in the current VES should resolve the issues - right?

... more to come ...

[phax]

@antoine-rehm would you be open to a call? If so, please send me a quick email, as I send it is more efficient to discuss this rather then to write all options here. If we post the outcome, it will do the trick I guess

[antoine-rehm]

Thanks, the BT-24 point is something I genuinely got wrong, and your quote from EN 16931 makes it clear: a user specification is supposed to identify itself in BT-24. And I share your concern about XML-only processing: a document that cannot state which ruleset governs it isn't self-contained, which is exactly the problem I ran into.

One thing worth adding, because it sharpens where the gap actually sits. The French spec does give itself an identity for the pure syntaxes — urn.cpro.gouv.fr:1p0:extended-ctc-fr appears in BT-24 for CII and UBL, and it also surfaces in the Peppol document type identifier, so both the payload and the transport layer can discriminate. Factur-X is the one case where neither can: BT-24 is pinned to the Factur-X URN by the Factur-X spec and by the PDF/A-3 XMP fx:ConformanceLevel, and a hybrid PDF carries no doctype either. So the reform gave itself a discriminator everywhere except the hybrid case — which is the case most French SMEs will actually send.

On /dd_and_validate: your criticism is fair and I won't argue it. I use it as a convenience for "tell me what this looks like", not as an authority on what rules apply.

On replacing the XSD in the current VES

I think your suggestion is the right one, and I'd like to withdraw the composite-VES idea in favour of it.

My initial reasoning was that the two rule sets aren't interchangeable. Comparing FACTUR-X_EXTENDED.sch (ZF 2.5) with 20260630_EXTENDED-CTC-FR-CII-V1.4.0.sch: 188 vs 856 distinct rules, 165 in common. Factur-X EXTENDED restates 13 EN 16931 rules as conditional BR-FXEXT-* variants — BR-22, BR-23, BR-24, BR-26, BR-38, BR-44 and BR-CO-04/10/11/12/13/15/16 — which CTC-FR keeps strict. For instance BR-22 in CTC-FR is the unconditional "Each Invoice line (BG-25) shall have an Invoiced quantity (BT-129)", while BR-FXEXT-BR-22 makes it conditional on the EXTENDED-only BT-X-8.

But on reflection that difference is the point rather than an obstacle. extended-ctc-fr is not Factur-X EXTENDED; it is a distinct French profile that deliberately keeps the EN 16931 rules strict where Factur-X relaxes them. For an invoice under the French mandate, the French rule set is the one that applies — so validating a French Factur-X EXTENDED invoice against CTC-FR and having it flagged on BR-22 is correct behaviour, not a false positive. The BR-FXEXT-* relaxations are what Factur-X permits outside a French context.

So replacing the XSD in fr.ctc:extended-cii — generic CII D22B → Factur-X EXTENDED — looks like the right fix: it reuses an existing VES instead of multiplying one per country × profile, and it applies the rule set that actually governs the document. That is clearly better than what I proposed.

On the postal-address discriminator: agreed it is not as simple as it looks, and your point about Peppol BIS rules reading VAT / Tax Representative data is something I had not considered. Happy to drop the country heuristic and rely on explicit caller-side selection instead.

[phax]

The brief outcome of the call was as following:

• The French CTC (Flux 2) VESID for CII will be using the respective Factur-X Extended CII XSDs matching the version that was active at the point of release (1.4.0-03 matching 1.0.9-2 and 1.4.0 matching 1.0.9 etc.).
• phorm will receive some form of note, that in the scenario of France, not all document type detections will be 100% accurate, as France made some bad design decision to not properly define BT-24