XML Format

[ChrisDeBmw]

We have to create a generic XML Format, I thought that we use the known parameter name as a unique identifier and then add all version-specific things as ChildNodes in the XML-Format.

[kmalinich]

I like the idea.

TunerPro is pretty close to XML as it sits, and there's many ways to programmatically parse XML (I'm partial to node.js).

We could read the version-specific TunerPro XDFs, match the parameter names together, then have the version-specific offsets under each parameter. It would really drastically reduce management overhead of the repo (i.e., if you need to update the XDF header, parameter description, formula, min/max, data type, decimal places, categories, etc., as it sits, you have to do it across every file)

Plus, just looking at the combined file might help paint a picture of differences between versions. I've had to scavenge from MSS54 and MSS54HP across the full range to help create my MSS52 XDF.

When creating the file, we could also add additional fields/keys that don't exist in the TunerPro format - like the additional/debug data in each description field for the output from the original XDF generator could be split out.

You could then re-generate the version-specific TunerPro XDFs from that file. Might also include options to not include things like debug data.

Since the only ordering methodology TunerPro recognizes for categories and parameters is the literal order they are in the file, and has no support for dynamic ordering like alphabetical or by binary offset, we might include XDF generation options for those, since it would be easy to do programmatically but is a massive pain to do manually (I've done it - since I'm fairly OCD about those kinds of things)

As far as the "master" file being XML - any particular reason you're partial to that format? Versus something like JSON or YAML, that is.

[ChrisDeBmw]

I’m currently developing a app written in C# which would parse the xdf, and merge them to a generic format. Each sw-version can override the parameters of the parent node. The generic format is also in XML, that is „easy“ to serialize/deserializer with .NET Framework :) Am 20.08.2019 um 15:16 schrieb Ken Malinich <notifications@github.com<mailto:notifications@github.com>>: I like the idea. TunerPro is pretty close to XML as it sits, and there's many ways to programmatically parse XML (I'm partial to node.js). We could read the per-version TunerPro XDFs, match the parameter names together, then have the per-version offsets under each parameter. It would really drastically reduce management overhead of the repo (i.e., if you need to update the XDF header, parameter description, formula, min/max, data type, decimal places, categories, etc., as it sits, you have to do it across every file) Plus, just looking at the combined file might help paint a picture of differences between versions. I've had to scavenge from MSS54 and MSS54HP across the full range to help create my MSS52 XDF. As far as the "master" file being XML - any particular reason you're partial to that format? Versus something like JSON or YAML, that is. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#9?email_source=notifications&email_token=AL7Q7IOHNRKJCHXVROFQ6ZLQFPVDHA5CNFSM4ILLX6GKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4WH5IQ#issuecomment-523009698>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AL7Q7ILFBNJLU6BXVGHVA43QFPVDHANCNFSM4ILLX6GA>.

[saildot4k]

Threw this out there as it may be a better method especially if one has to fix offsets as that will still be labor intensive:
http://www.m3forum.net/m3forum/showthread.php?p=1067986240#post1067986240

Curious if this is possible or to what extent it may be possible and how:
Can any HW variant be updated to any similar family OS IE MSS54HP 2501 to MSS54 2701?
Not that I need it, but it was talked about years ago in the Comprehensive thread, and someone on the github project is parsing all the XML data so that if we fix something simple, such as a Tunerpro conversion, we do not need to edit all XDFs by hand.
But if we can get users to instead just upgrade to say MSS54 2801 and MSS54HP 2701, then really only need to upkeep 1, especially for offset fixes.

[saildot4k]

From Terra:

Any MSS54 can run any MSS54 software and any MSS54HP can run any MSS54HP software.
Complication is 2701 and 2801 Euro versions don't exist, so we'd need to support more like 5 versions (Euro / non-Euro in HP/non-HP + CSL)

[saildot4k]

I think this would be the way to move forward: get rid of what we don't need, maybe even write up/document how to upgrade so that we have less XDFs for the tool to parse and help us fix.

[ChrisDeBmw]

So i can report some progress, i can merge xdf-parameters (till now only constants) together.
Here is an example of my format till now:

<parameter guid="1f5d7f5e-0ac0-4f5d-806a-6cd48aedfeb6">
     <id>K_MD_SK_NHYS</id>
    <addedByVersion>0401</addedByVersion>
    <DateCreated>2019-08-25T20:13:31.834282+02:00</DateCreated>
    <DateModified>2019-08-25T20:13:31.834284+02:00</DateModified>
    <xdfParamType>XDF.XDFCONSTANT</xdfParamType>
    <XDFCONSTANT uniqueid="0x895" vislevel="10">
      <title>K_MD_SK_NHYS</title>
      <description>K_MD_SK_NHYS  950A&amp;#013;&amp;#010;HW:108 Version:211323002001J424&amp;#013;&amp;#010;Created by find routine V0.35&amp;#013;&amp;#010;BMWFlash/Galleto File Orientation [Slave Master]&amp;#013;&amp;#010;Matched Length: 11  MatchRatio: 1.00&amp;#013;&amp;#010;BlockFound: 0 NearbyBlockMatch: 19 Var: 2197 HasDuplicate?: No &amp;#013;&amp;#010;Units- "Upm"  &amp;#013;&amp;#010;Function: SK-Momentenmanager  SizeChange: 0 &amp;#013;&amp;#010;</description>
      <CATEGORYMEM index="2" category="58" />
      <EMBEDDEDDATA mmedaddress="0x950A" mmedelementsizebits="16" mmedmajorstridebits="0" mmedminorstridebits="0" />
      <units>"Upm"</units>
      <datatype>0</datatype>
      <unittype>0</unittype>
      <DALINK index="0" />
      <MATH equation="x">
        <VAR id="x" />
      </MATH>
    </XDFCONSTANT>
    <xdfParamVersionOverrides versionId="2001">
      <XDFCONSTANT>
        <description>K_MD_SK_NHYS  950A&amp;#013;&amp;#010;HW:108 Version:211323002001J424&amp;#013;&amp;#010;Created by find routine V0.35&amp;#013;&amp;#010;BMWFlash/Galleto File Orientation [Slave Master]&amp;#013;&amp;#010;Matched Length: 11  MatchRatio: 1.00&amp;#013;&amp;#010;BlockFound: 0 NearbyBlockMatch: 19 Var: 2197 HasDuplicate?: No &amp;#013;&amp;#010;Units- "Upm"  &amp;#013;&amp;#010;Function: SK-Momentenmanager  SizeChange: 0 &amp;#013;&amp;#010;</description>
        <EMBEDDEDDATA mmedaddress="0x950A" mmedelementsizebits="16" mmedmajorstridebits="0" mmedminorstridebits="0" />
      </XDFCONSTANT>
    </xdfParamVersionOverrides>
  </parameter>

So the override section can override every xml-node in the parent node which is version depend.
These overrides are then used with the parent node to generate a xdf file.

Any ideas how we should handle the categories?
Best approach would to use a general store, valid for all versions we want to produce.
Otherwise we have to save all categories for all versions.

[saildot4k]

Curious, what is your end goal with this?

Another update from Terra

Alternatively, there are US software variants that match the euro program versions, they just need to be shared (not present in current releases of progman)

So with that, why not consolidate the XDF parser tool when we can consolidate it to 5 or less?

[ChrisDeBmw]

Hm, for example, i bought my car with a mapped 2001 version but currently im running 0401 (csl) on my ecu.
So my goal is to make a generic parser/storage which can support every version that exists out there.
Goal of the format is to iterate through all parameters, and override the XDF-nodes parameters with these in the override section for the requested program-version XDF.

For me that is now little more work to do but later we can generate as many XDFs we want :-)

[ChrisDeBmw]

Sorry guys but im currently very busy.. but this project is always in my focus :-)

[saildot4k]

Not a problem at all. Feel free to add some organization for your program so that others may help with the code if they get involved, or if you feel, make your own repository and invite others, however you want to proceed.

[kmalinich]

I'm still thinking about how this would work.

Also, not to add complication.. but as an E39 M5 owner, I'd kind of feel bad leaving MSS52 out in the cold.

And on the note of categories - when working on my (incomplete) XDF for my M5, I went in a pretty different direction compared to any XDFs (for any make/model/engine/transmission) that I've seen - maybe load it up in TunerPro to see if it stimulates any ideas.

[ChrisDeBmw]

@kmalinich my program i am working on will be a generic xdf-format parser/generator. This should then also work for a MSS52.
First step is to build a parser which can concatinate all ecu-versions into one xml file.
Second step is generating new xdf with corrected parameters for the specific ecu versions.

So far only the XDFTables parser is missing, XDFConstants and XDFFunctions are working.
Here is a XDFFunction example, added by ecu 2001 and with a specific part for 0401 which overrides some data:

<parameter guid="f3360633-157c-47e8-bf09-1f4524a4b6d5">
    <id>KL_EDK_VORST</id>
    <addedByVersion>2001</addedByVersion>
    <DateCreated>2019-10-08T21:13:22.6754729+02:00</DateCreated>
    <DateModified>2019-10-08T21:13:22.6754729+02:00</DateModified>
    <xdfParamType>XDF.XDFFUNCTION</xdfParamType>
    <XDFFUNCTION uniqueid="0xB2" vislevel="10" flags="0x0">
      <title>KL_EDK_VORST</title>
      <description>KL_EDK_VORST  8146&amp;#013;&amp;#010;HW:108 Version:211323002001J424&amp;#013;&amp;#010;Created by find routine V0.35&amp;#013;&amp;#010;BMWFlash/Galleto File Orientation [Slave Master]&amp;#013;&amp;#010;Matched Length: 41  MatchRatio: 1.00&amp;#013;&amp;#010;BlockFound: 0 NearbyBlockMatch: 20 Var: 178 HasDuplicate?: No &amp;#013;&amp;#010;Units- X: "%" Y: "%tv"  &amp;#013;&amp;#010;Named- X: edk_s Function: EDK  SizeChange: 0 &amp;#013;&amp;#010;</description>
      <CATEGORYMEM index="2" category="24" />
      <XDFAXIS id="x" uniqueid="0x0">
        <EMBEDDEDDATA mmedaddress="0x8132" mmedelementsizebits="16" mmedcolcount="10" mmedmajorstridebits="16" mmedminorstridebits="0" />
        <units>"%"</units>
        <indexcount>10</indexcount>
        <embedinfo type="1" />
        <datatype>0</datatype>
        <unittype>0</unittype>
        <DALINK index="0" />
        <MATH equation="x/10">
          <VAR id="x" />
        </MATH>
      </XDFAXIS>
      <XDFAXIS id="y" uniqueid="0x0">
        <EMBEDDEDDATA mmedaddress="0x8146" mmedelementsizebits="16" mmedcolcount="10" mmedmajorstridebits="16" mmedminorstridebits="0" />
        <indexcount>10</indexcount>
        <decimalpl>1</decimalpl>
        <embedinfo type="1" />
        <datatype>0</datatype>
        <unittype>0</unittype>
        <DALINK index="0" />
        <MATH equation="x/20">
          <VAR id="x" />
        </MATH>
      </XDFAXIS>
    </XDFFUNCTION>
    <xdfParamVersionOverrides versionId="2001" />
    <xdfParamVersionOverrides versionId="0401">
      <XDFFUNCTION>
        <description>KL_EDK_VORST  8146&amp;#013;&amp;#010;HW:211 Version:211325000401PD11&amp;#013;&amp;#010;Created by find routine V0.55&amp;#013;&amp;#010;BMWFlash/Galleto File Orientation [Slave Master]&amp;#013;&amp;#010;Matched Length: 41  MatchRatio: 1.00&amp;#013;&amp;#010;BlockFound: 0 NearbyBlockMatch: -1 Var: 178 HasDuplicate?: No &amp;#013;&amp;#010;Units- X: "%" Y: "%tv"  &amp;#013;&amp;#010;Named- X: edk_s Function: EDK  SizeChange: 0 &amp;#013;&amp;#010;Description: None &amp;#013;&amp;#010;&amp;#013;&amp;#010;</description>
      </XDFFUNCTION>
    </xdfParamVersionOverrides>
  </parameter>

For the moment i will save every ecu versions category-definitions in the generic xml.
I have attached a XML from the current development state with ecu 2001 as basis and version 0401 added. Note: XDFTables are not included!
Rename txt to xml.
ecuparams.txt

[saildot4k]

Any updates?

[ChrisDeBmw]

Hey guys im sorry but i lost the focus at this project...
I just sent an message to the developer of tunerpro requesting his code for xdf-parsing, maybe he will provide me something. That would help me a lot.
Will try to get thing on now :-)

[saildot4k]

Thanks for the update! I can't blame you for losing focus. BTW if anyone has anything to add to the wiki, please do so. As it seems m3 forum is down, maybe a synopsis of how tables play together such as when certain vanos and throttle maps are applied....

[ChrisDeBmw]

I did some work on this thing after the answer from the tunerpro developer which cannot help me providing some code.
So my current status is:

• merging all xdf versions into one big storage works
• generate particiular xdfs from this storage works

Merging all together produces a 30MB XML-File.. but its the easiest way for editing i think.
A real database would need a gui-tool for editing and so on.
Many things are doubled, so in every version a parameter has its own description because the description contains the version-number and other unique data.
These things blows up the storage.

I would suggest that we start with one big storage and release xdf-packages once we think its time for. Otherwise we will flood the internet with xdf versions and nobody can rely on a versionnumber.
What do you guys think?

[saildot4k]

I am not sure to be honest, but I think maybe it could be wise to start a thread on m3forum.us. I think you could get more feedback there in the nearer future?