Find XSD (or other) definition of EDIFACT format

[nerdoc]

I am looking for an official, machine readable document that describes the edifact format, so pydifact can parse it and validate the Segments it reads in against it. I have already written to the UN, but I get no answer besides the documents on unece.org - which are written for humans. No XSD schemata, no JSON formats, nothing. Just plain prosa, and tables. Maybe I'll open an issue for that, if anyone wants to join there.

General information

https://unece.org/trade/uncefact/unedifact/part-4-chapter-22-syntax-rules
https://unece.org/fileadmin/DAM/trade/edifact/untdid/d422_s.htm

Data elements

https://service.unece.org/trade/untdid/d22b/tred/tredi1.htm

There are codes listed in separate pages like this
But I can't find a generic description/list in machine readable format.

[lord-haffi]

I have already written to the UN, but I get no answer besides the documents on unece.org - which are written for humans. No XSD schemata, no JSON formats, nothing. Just plain prosa, and tables.

Haha, before this year this was also the case for the entire market communication of energy companies in germany. "Luckily", now we can buy corresponding XML-files. I feel your pain though.

If the data source i.e. the website is more or less stable you can try to scrape it. But it's definetely a hell lot of work. We did it with docx-files. It was pain in the a**. Sadly, I don't think you could reuse much of the code we wrote since we scraped an entirely different data source. But if you're interested or could need some design inspiration you can take a look at our open source project.

[lord-haffi]

Btw, me and a colleague are currently building such a validation software (for EDIFACT's in the energy market communication) based on the new XML-files. It will be commercial software though. But if you manage to create / get a machine readable data source, I can give you some design ideas of our ansatz if you want.

[nerdoc]

Sure, that would help in any case! Scraping could be an option, for the begin I threw it at ChatGPT and let it create a JSON file. But the LLMs all dream too much and have no stable output. There is also bots that seems to be on the "market" for a long time. But even there is only one xsd file that seems to be generated somehow, but only for a minimal subset of EDIFACT.

It's always the same. Why the heck am I the only/first one that needs this? EDIFACT is here since 1988 - and NOONE EVER had the idea of using a formal description of the format? machine readable, reusable, standardized? Nope. Sigh.

I think the data sources are quite stable, so I'd want to try scaping it maybe. But my time ATM is limited ;-)

[lord-haffi]

I don't quite understand yet which kind of document you are looking for. Do you mean some kind of document type for EDIFACT like XSD is for XML? So something like "Next segment shall be XY with data elements AB..."?
If yes, sorry to disappoint you, but that doesn't exist. Yet. As I said, recently the BDEW published XML-files which act as an XSD for EDIFACT (energy market communication in germany). Our usecase seems to be a bit more complex though as we have two documents defining not only the syntactical appearance of the EDIFACT but also some semantics and conditions when which element with which content has to be defined and so on.

In your case, you can of course scrape the tables quite easily. But interpreting the prosa inside is basically impossible (in a robust way). We also have this kind of prosa still present inside the conditions. And we are currently hard coding these. There is not really a way around it. We also tried using LLMs but the results were really bad and it was actually more work to generate and post correct the code from LLM than just hard coding it directly.

About bots you posted... last commit on master 8 years ago? I think it's dead ^^

[lord-haffi]

To describe our ansatz of the validation software:

1. Parse the BDEW XML-files into JSON. Because XML sucks ^^ I will refer to this as the "XML"-files in the following but I actually mean the object structure resulting from the JSON.
2. Parse an EDIFACT into Interchange using your project.
3. Use the XML-files to build a tree structure from the Interchange. This message tree structure contains segment groups, segments, data element groups and data elements. Truncated data elements will automatically be "re"-added by comparing to the XML-file. This makes the validation easier later on.
4. Do validation on the tree structure. We do two different validations for both XML-files as errors result in a different error EDFACT format. The first one is called CONTRL, the second is APERAK.
   4.1. The CONTRL-validation comes first but is not included in our current scope. It's way easier to implement though.
   4.2. For the APERAK validation we iterate through our message tree structure and the XML-structure in parallel to account for repeated elements as well as for missing elements (which will only be present in the XML). Our iterator function is... well, I did prettier things in my life. But it works ^^ We also have conditional expressions which are somewhat parsable plus the unparsable condition codes which are described with prosa. However, these conditions will be evaluated to decide if the corresponding (potentially missing) element is correct.
5. The errors will be collected, serialized and returned as JSON through a web API.

[nerdoc]

Yes, I meant some XSD files. And I know XMLS sucks, for myself I played around a bit with creating JSON definitions of EDIFACT segments parsed from the official docs on the UNECE site.
This looks like this for service codes:

{
  "0001": {
    "name": "Syntax identifier",
    "description": "a3, upper case, Controlling Agency (e.g. UNO = UN/ECE) and a1 stating level (e.g. A) (which together give UNOA)",
    "mandatory": true,
    "repr": "a4",
    "codes": {
      "UNOA": {
        "name": "UN/ECE level A",
        "description": "As defined in the basic code table of ISO/IEC 646 with the exceptions of lower case letters, alternative graphic character allocations and national or application-oriented graphic character allocations."
      },
      "UNOB": {
        "name": "UN/ECE level B",
        "description": "As defined in the basic code table of ISO/IEC 646 with the exceptions of alternative graphic character allocations and national or application-oriented graphic character allocations."
      },
      // etc.
      // Oh, i forgot, JSON does not allow comments. OMG, what a dumb structure in 2025.
    }
  },
  "0002": {
    "name": "Syntax version number",
    "description": "Version number of the syntax.",
    "mandatory": true,
    "repeat": 1,
    "repr": "n1",
    "codes": {
      "1": {
        "title": "Version 1",
        "description": "ISO 9735:1988."
      }
    }
  },
  "0004": {
    "name": "Interchange sender identification",
    "description": "Code or name as specified in IA.",
    "mandatory": true,
    "repeat": 1,
    "repr": "an..35"
  },
  // ...
}

And then for e.g. the UNB segment:

{
    "UNB": {
        "name": "Interchange header",
        "description": "To identify an interchange.",
        "notes": [
            "S001/0002 shall be '4' to indicate this version of the syntax.",
            "The combination of S002, S003, and 0020 uniquely identifies the interchange."
        ],
        "elements": [ "S001", "S002" ]
    }
}

We have to do that for each EDIFACT version, as they differ minimally. I ran into that problem, as I have plenty of edifact files that have a "wrong" datetime format in their UNB segment - here is my v4 variant.

            "S004": {
                "name": "Date and time of preparation",
                "description": "Date and time of preparation of the interchange.",
                "elements": {
                    "0017": {
                        "name": "Date",
                        "description": "Local date when the interchange was prepared (CCYYMMDD).",
                        "mandatory": true,
                        "repeat": 1,
                        "type": "n8"
                    },
                    "0019": {
                        "name": "Time",
                        "description": "Local time when the interchange was prepared (HHMM).",
                        "mandatory": true,
                        "repeat": 1,
                        "type": "n4"
                    }
                }
            },

but my data look like this:

UNB+ANSI:1+ME123456+ME654321+180312+1623+1'UNH+0001+MEDRPT:1:901:UN'BGM+012+++20180312++211903101965+19651003'FTX+BFD++

You can see that the service code 0017 looks after a "n8" date in the form CCYYMMDD - but my EDIFACT file provides its date in the UNB segment as YYMMDD - I first thought this is wrong, until I found out that we in Austria are using the EDIFACT format from 1990 (WTF!?), from our description it is "EDIFACT DIR 90.1" - which is EDIFACT v1 in the sense of SYNTAX IDENTIFIER.

So I think I need JSON files like these above for each EDIFACT version.

And, in e.g. the v4/UNB.json, I can't really write "elements": [ "S001", "S002" ] - because in the e.g. segment UNB of a certain message type, the definition "may" override the global service codes as well, so EDIFACT format 40100 could say, in the XZY message type, the ABC segment has the usual 0023,0043,0017 elements, BUT, in 0017, the type is not n4, but n2.

Which is really nasty, and I cant write "elements": [ "S001", "S002" ], but i have to re-define them in JSON again - because just "Links" to S001 are not enough.

Pfffffff.

[lord-haffi]

Sorry, for the late response. I think I understand your problem now. Unfortunately, I see no simple (automated) solution for your use-case. It seems like you would have to create multiple scrapers and fiddle the stuff together you need. But I think you may are better to go with some hand written files if the changes are rare enough.

Your JSON-format looks actually very similar to the definitions of the BDEW. But in the BDEW-documents they just scramble it together in one big file i.e. in fields like elements are actually the full objects not just references. I think they chose this way because the IDs they use are no real IDs. There can be multiple different segment groups or even data elements with the same ID. Please don't ask me why. I would throw some rocks through their windows myself if I could :D Anyway, I hope this is not a thing for your use-case and you don't have to worry about such things.

Conclusion:

Unfortunately, I have no easy automated solution for you. I think it would be possible (with a lot of work) to scrape the tables. But no chance for the prosa. If you really need that, you have to write it by hand.
The JSON-structure you posted looks good. Just look out for problems like non-unique "IDs" to prevent design errors. My advice: Double-check everything. Even if it's the most obvious thing in the world. Like "IDs"...

I don't think you have to create documents for each version though. I would probably create only two files for now (current version and yours) and leave the rest for the future. Until someone creates an issue for example ^^

PS: EDIFACT from 1990 is a great example of "historisch gewachsen" :D It's wild. Will definetely tell my colleagues about this ^^

[nerdoc]

Thanks for the hint. I'm just working on the feature in a way you already recommended: Writing code per hand. Or with a little help of AI.
I think that opening a huge .json file is not the way I feel comfortable when running pydifact. And there are approx. 80% the same fields from EDIFACT v1 to v4. With code, I can inherit class UNBSegment v4 from v3, v1 etc. In .json files, I'd have to do that inheriting logic additionally. NOT.
So I decided to do it this way (in pydifact.syntax.v4)

__version__ = 4
class SyntaxIdentifier(v1.SyntaxIdentifier):
    code = "0001"
    repr = "a4"

...
class CSyntaxIdentifier(v1.CSyntaxIdentifier):
    code = "S001"
    title = "Syntax identifier"
    schema = [
        (SyntaxIdentifier, M, "a4"),
        (SyntaxVersionNumber, M, "an1"),
        (ServiceCodeListDirectoryVersionNumber, C, "an..6"),
        (CharacterEncoding, C, "an..3"),
    ]
...
class UNBSegment(v3.UNBSegment):
    """Interchange header."""

    tag = "UNB"
    version = __version__
    codes = [
        (CSyntaxIdentifier, M, 1),
        (CInterchangeSender, M, 1),
        (CInterchangeRecipient, M, 1),
        (CDateAndTimeOfPreparation, M, 1),
        (InterchangeControlReference, M, 1),
        (CRecipientReferencePassword, C, 1),
        (v1.ApplicationReference, C, 1, "an..14"),
        (v1.ProcessingPriorityCode, C, 1, "a1"),
        (v1.AcknowledgementRequest, C, 1, "n1"),
        (InterchangeAgreementIdentifier, C, 1, "an..35"),
    ]

And I use the existing factory to create the segments from the parsed data.

Yes, the "ids" like 0001, 0002 etc. are completely useless. I don't know what the EDIFACT creators thought about that. Especially when implementations of these data types can vary from segment to segment, both using the same "id" field.

But ok. It was 1990, when a 80286 was a high end computer ;-)

I think that approach will do well, I have a partly working prototype here in another branch and just want to check if the code is working well.

[nerdoc]

Oh, and by the way: I will implement service codes that I need - feel free to add yours then as PRs.
Next step would be messages, one level higher, in the same manner.

[nerdoc]

The first part is already in master, and decision is made how it is implemented. Code, not data. Maybe it could be done better, but it works well and could be changed without breaking the API later. So I'll close this issue for now.

[nerdoc]

Just one thing @lord-haffi - I'd like to read your feedback here (or anyone else's):

I implemented a basic structure for Segments like UNBSegment(...) that can validate() each segment, which is done automatically in the factory.
I don't know if this is a good idea, as it may break existing, working setups, especially when data are received that are non-standard.

Could you test the code - any feedback welcome.

[lord-haffi]

Could you test the code - any feedback welcome.

I'll definetely do! The master branch, right? I think, I'll have time in the upcoming week to take a look at it.
Thanks a lot for your work!

Have a nice weekend :)

[nerdoc]

Yes, master branch!