-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
XML to JSON conversion (OpenEpcis, Mimasu, EpcisWorkbench, or JXON) #276
Comments
FWIW the JSON Schema are now in process of being modularized.
…On Mon, May 17, 2021 at 12:56 AM Vladimir Alexiev ***@***.***> wrote:
While the XSD and JSON schemas (should) define EPCIS content precisely,
and the spec and UML makes sure the same content is expressed, there is no
formal definition of a transformation between XML and JSON (and back). Such
transformation would be a valuable dev resource.
@mgh128 <https://github.com/mgh128> mentioned JXON in #237
<#237>
https://www.balisage.net/Proceedings/vol7/html/Lee01/BalisageVol7-Lee01.html
. I looked at it and it seems perfect for the job. Any volunteers to
develop the JXON transformation and validate it?
This will also allow us to make XML for every JSON example (there's
another task here suggesting this).
BTW, JXON can be attached to either XSD or RNC schemas. The example
"equivalent schema in XSD using Annotations instead of comments" shows how
much nicer is RNC compared to XSD. @CraigRe <https://github.com/CraigRe>
@RalphTro <https://github.com/RalphTro> is there any chance the group
could adopt RNC as XML master schema? I can convert XSD to RNC.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#276>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AQEZJLTVOIJ2LU5HJXDHBOLTOBETBANCNFSM447LB4RQ>
.
--
*IOTA Foundation*
c/o Nextland
Strassburgerstraße 55
10405 Berlin, Germany
Board of Directors: Dominik Schiener, Serguei Popov, Navin Ramachandran
ID/Foundation No.: 3416/1234/2 (Foundation Register of Berlin)
|
While working on #231, one of the examples Example-TransactionEvents-2020_07_03y.xml failed to convert at https://www.mimasu.nl/epcis/xmljson, maybe because it has @RalphTro @CraigRe Do you know anyone from Mimasu or anyone using custom extensions often? |
@dannyhaak runs Mimasu; probably quite a few deployments use implementation-specific extensions... |
@dannyhaak https://www.mimasu.nl/epcis/xmljson fails on https://github.com/VladimirAlexiev/EPCIS/blob/issue-231/XML/Example-TransactionEvents-2020_07_03y.xml with That file was in the original examples but:
@sboeckelmann could you validate that file with the fixed XSD in your PR? |
@VladimirAlexiev it's valid: |
@sboeckelmann Please validate against |
sorry - my mistake Makefile still references the master data xml. But it's also valid for EPCglobal-epcis-2_0.xsd Once we have a nicer master containing latest PRs I will revisit the Makefile and optimize. |
|
@CraigRe could you assign to @dannyhaak to see why xmljson fails on https://github.com/VladimirAlexiev/EPCIS/blob/issue-231/XML/Example-TransactionEvents-2020_07_03y.xml ? |
@VladimirAlexiev , negative. Closing this topic now, as it is DEFERRED. |
@dakbhavesh wrote in #286 "While building XML <-> JSON converter..." Currently we don't have a XML->JSON "reference implementation" that works in all cases.
|
Dear @VladimirAlexiev, the XML <-> JSON converter is still work in progress. Once it is in good shape we will definitely make the service available for all to use and going to make it open source as well. To make it open source, there is a little bit of work involved in extracting it as an independent component from others. However, we will get to it as well. |
@dakbhavesh does it convert the cited example? |
Hi @VladimirAlexiev, The example showcase a classic case of paradigm mismatch between JSON and XML world. The mismatch: XML allows repetition of elements i.e. rail:vehicle elements are repeating without any wrapper. JSON wouldn't allow duplicate keys. The best converter would be able to do (for now) is to translate such top-level repeating elements into a JSON array like below (notice that array name is exactly the same as element name in XML)
@VladimirAlexiev - What do you think? Any better ideas? |
@dakbhavesh This is the only idea that makes sense! Does that mean one cannot make a faithful convertor without knowing a bit about the extension schema (namely field cardinalities)? The convertor can examine the XML instance and emit an array only if it finds several copies of the same element, but that would not be the same JSON representation as the case of a single element. Ps: there is no such problem in RDF because a single element and an array if elements are represented in the same way. |
@VladimirAlexiev - You're right that without providing hints converter will assume array / single object based on a number of elements which will be inconsistent. However, this seems to us more of a business alignment. Meaning, at the business level it should be enforced that array elements should have a wrapper in XML based event. If that can be agreed on then we don't need and complexity of providing hints (about datatype of elements) and keep the converter as simple as possible. |
Hi @VladimirAlexiev, |
Sven on 16 Jul: For those of you who don’t care that much about web frontends, you can also call the service directly:
We are looking forward to your feedback which will allow us to improve it! Ralph: Dear Sven and Team, This is really cool! Thanks a lot for this initiative. I just gave it a couple of tests, most went well.
Kind regards and once again a BIG THANK-YOU Sven:
Thank you for all the nice feedback! It helps us to improve and stabilize the tools and the repo. |
@sboeckelmann, @shalika-singh @dakbhavesh
Convertor bugs:
Convertor improvements:
This strikes me as an excellent way to unify our XML and JSON examples! Thanks a lot! |
Aravinda Baliga B commented:
|
Issue 6
|
"@context": [
"https://gs1.github.io/EPCIS/epcis-context-simple.jsonld",
{"rail": "urn:gs1:epcisapp:rail:",
"rail:vehicleMasterGIAI": {"@type": "@id"},
"rail:trainAxleCount": {"@type": "xsd:integer"},
"rail:vehicleAxleCount": {"@type": "xsd:integer"},
"rail:vehiclePosition": {"@type": "xsd:integer"},
"rail:vehicleUniquelyIdentified": {"@type": "xsd:boolean"}
}
] |
Thanks a lot for testing and providing feedback on the tools. We have already incorporated some of the changes and some are in development. Please find the relevant comments below: Issue 5: Issue 6: During the development of the conversion tool, the goal was to develop straight forward conversion tool without check constraints on business logic. Hence, the conversion is pretty much one-to-one. For now, "AggregationEvent" with the "Delete" action, addition of the empty array will keep on low priority as its specific to particular eventType. If a lot of community members are looking for these types of requirements and if it's necessary then we will implement it along with another such type of requirements. Issue 7: Issue 8: Issue 10: Issue 11: However unable to understand the second part of the issue, can you please provide some more explanation on this?
In addition, the following are the new features available in the next deployment:
|
@Aravinda93 @sboeckelmann @dakbhavesh
|
Thanks a lot for taking your time to test and providing this valuable feedback. We have started looking at these suggestions and started fixing them. We will make the appropriate changes and update you soon. |
Decision 21 Sep:
|
Thanks a lot for all your feedback. Please find the comment related to the issues mentioned above and discussed on the 21.09 GS1 call. Issue 12: (Fixed). Issue 6a (Fixed): Issue 8 (Fixed): Issue 10 (Fixed/Reverted): Issue 11 (Fixed - available in next deployment): Issue 12 (Fixed): Issue 13 (Pending): Issue 14 (Fixed): Issue 15 and Issue 16 (Pending): Issue 17 (Fixed): |
lines 81-116 of
https://github.com/gs1/EPCIS/blob/master/JSON-Schema/schemas/AggregationEvent-JSON-Schema.json
AND
lines 77-106 of
https://github.com/gs1/EPCIS/blob/master/JSON-Schema/schemas/AssociationEvent-JSON-Schema.json
already require any (at least one) of the following to be true for an event
to be valid:
- childEPCs is required to be a non-empty array
*OR*
- childQuantityList is required to be a non-empty array
*OR*
- action='DELETE'
If we were to make childEPCList mandatory (a "required" property,
alongside "action" ) in JSON-Schema (even though it could be an empty
array), it might help the XML-->JSON-LD conversion but it would also mean
that we would always need to express "childEPCs" : [] even within events
that have a populated childQuantityList or where "action" : "DELETE" even
in situations where neither childEPCs nor childQuantityList are specified.
Is that really the change we want to make?
Wouldn't there also be a corresponding need to require childQuantityList to
be 'required' in the JSON Schema for the same reason of making the XML-->
JSON-LD conversion easier?
…On Tue, Sep 21, 2021 at 3:57 PM Vladimir Alexiev ***@***.***> wrote:
Decision 21 Sep:
- Issue 6a: make childEPCList mandatory in JSON-Schema (though can be
empty array).
@jmcanterafonseca-iota <https://github.com/jmcanterafonseca-iota> ,
@dakbhavesh <https://github.com/dakbhavesh> , @mgh128
<https://github.com/mgh128> to implement
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#276 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABSXRL36VUVUXTOX4MAG4WLUDCMPDANCNFSM447LB4RQ>
.
|
Issue 11: what library do you use for http access? It should support
redirects.
Eg in curl (a command line tool) the `-L` option does that.
|
@VladimirAlexiev |
Issue 6a: the important decision is to make JSON Schema and XSD correspond precisely, so the tool won't have to make empty elements. Issue 8a (new) If the result is an error then DO USE word wrapping in the document textbox. Issue 18 (new, CRITICAL): EPCIS context URL is missing from JSON.
Issue 19 (new): https://raw.githubusercontent.com/gs1/EPCIS/master/XML/WithSensorData/SensorDataExamples.xml Issue 13 (worse): JSON-simple-context/WithSensorData/SensorDataExample17.jsonld has
Issue 20 (enhancement idea): run XSD respectively JSON validation on the resulting document
I validated issues 4, 7, 8, 12, 14. |
Yes, |
Thanks a lot again for taking your time to check the tool and for providing your valuable feedbacks. Please find below our comments related to each specified issue: Issue 6a: Issue 8a: Issue 18: Issue 19: Issue 13: Issue 20: For further improvement, we have integrated CodeMirror editor into our application. CodeMirror beautifies the syntax for XML/JSON_LD also it will highlight the syntax errors in the respective area. We feel this is making our application much appealing and easy to use. Also, we have made many other minor changes to the code for optimization purposes and better performance. |
Issue 6a: As @mgh128 wrote, both childEPCs and childQuantityList should be optional because they are alternatives to each other. The best approach is to ensure the two schemas conform to each other, so the tool doesn't need to add any elements out of thin air. Issue 18: the official URL will be at gs1, but is not yet available. Issue 13: verified Issue 21: Thanks for CodeMirror syntax highlighting!
Issue 20a (syntax validation): I now notice you already have syntax validation embedded, great! But in both cases it's very functional, thanks! (Issue 20 is about schema validation) |
@VladimirAlexiev and @Aravinda93 as a reminder, all of this needs to be logged via the Community Review at: Thanks! |
@CraigRe Yes, if it's concerned with any file in the public review bundle. Not sure if this tool was within that. |
@mgh128 agreed and apologies for any confusion -- I just want to make sure we capture commrev comments as required. |
Thanks a lot again for your valuable feedbacks. Please find our response based on each of your described issues: Issue 6a: Issue 18: Issue 21:
Issue 20: |
hi @Aravinda93 @sboeckelmann @dakbhavesh the site seems to be down: |
Sorry for the inconvenience. But it’s up and available again
… Am 09.11.2021 um 10:25 schrieb Vladimir Alexiev ***@***.***>:
hi @Aravinda93 <https://github.com/Aravinda93> @sboeckelmann <https://github.com/sboeckelmann> @dakbhavesh <https://github.com/dakbhavesh> the site seems to be down:
both https://tools.openepcis.io/ <https://tools.openepcis.io/> and https://tools.openepcis.io/Documentconverter <https://tools.openepcis.io/Documentconverter>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#276 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE72UTQ7VO3G7RSWVBWAQJLULDSGHANCNFSM447LB4RQ>.
Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
While the XSD and JSON schemas (should) define EPCIS content precisely, and the spec and UML makes sure the same content is expressed, there is no formal definition of a transformation between XML and JSON (and back). Such transformation would be a valuable dev resource.
@mgh128 mentioned JXON in #237
https://www.balisage.net/Proceedings/vol7/html/Lee01/BalisageVol7-Lee01.html . I looked at it and it seems perfect for the job. Any volunteers to develop the JXON transformation and validate it?
This will also allow us to make XML for every JSON example (there's another task here suggesting this).
BTW, JXON can be attached to either XSD or RNC schemas. The example "equivalent schema in XSD using Annotations instead of comments" shows how much nicer is RNC compared to XSD. @CraigRe @RalphTro is there any chance the group could adopt RNC as XML master schema? I can convert XSD to RNC.
The text was updated successfully, but these errors were encountered: