Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

XML to JSON conversion (OpenEpcis, Mimasu, EpcisWorkbench, or JXON) #276

Closed
VladimirAlexiev opened this issue May 16, 2021 · 40 comments
Closed

XML to JSON conversion (OpenEpcis, Mimasu, EpcisWorkbench, or JXON) #276

VladimirAlexiev opened this issue May 16, 2021 · 40 comments
Labels
DEFERRED Issues addressed will not "break" 2.0 deployments, postponed for inclusion in next EPCIS revision. Implementation support

Comments

@VladimirAlexiev
Copy link
Contributor

While the XSD and JSON schemas (should) define EPCIS content precisely, and the spec and UML makes sure the same content is expressed, there is no formal definition of a transformation between XML and JSON (and back). Such transformation would be a valuable dev resource.

@mgh128 mentioned JXON in #237
https://www.balisage.net/Proceedings/vol7/html/Lee01/BalisageVol7-Lee01.html . I looked at it and it seems perfect for the job. Any volunteers to develop the JXON transformation and validate it?

This will also allow us to make XML for every JSON example (there's another task here suggesting this).

BTW, JXON can be attached to either XSD or RNC schemas. The example "equivalent schema in XSD using Annotations instead of comments" shows how much nicer is RNC compared to XSD. @CraigRe @RalphTro is there any chance the group could adopt RNC as XML master schema? I can convert XSD to RNC.
@jmcanterafonseca-iota
Copy link
Collaborator

jmcanterafonseca-iota commented May 17, 2021 via email

@VladimirAlexiev VladimirAlexiev mentioned this issue May 18, 2021
Open
@VladimirAlexiev
Copy link
Contributor Author

While working on #231, one of the examples Example-TransactionEvents-2020_07_03y.xml failed to convert at https://www.mimasu.nl/epcis/xmljson, maybe because it has rail: examples?
Then I found I can't validate it (#280).

@RalphTro @CraigRe Do you know anyone from Mimasu or anyone using custom extensions often?

@CraigRe
Copy link
Collaborator

CraigRe commented May 18, 2021

@dannyhaak runs Mimasu; probably quite a few deployments use implementation-specific extensions...

@CraigRe CraigRe changed the title JXON schema XML to JSON conversion (was JXON) May 18, 2021
@CraigRe CraigRe added DEFERRED Issues addressed will not "break" 2.0 deployments, postponed for inclusion in next EPCIS revision. Implementation support labels May 18, 2021
@VladimirAlexiev
Copy link
Contributor Author

@dannyhaak https://www.mimasu.nl/epcis/xmljson fails on https://github.com/VladimirAlexiev/EPCIS/blob/issue-231/XML/Example-TransactionEvents-2020_07_03y.xml with Unspecified error in parsing XML.

That file was in the original examples but:

  • had the rail: namespace repeated multiple times in sub-elements, I put it once on top.
  • also, made it explicit in rail:vehicle* elements.

@sboeckelmann could you validate that file with the fixed XSD in your PR?

@sboeckelmann
Copy link
Collaborator

@VladimirAlexiev it's valid:
xmllint --noout --schema ../XSD/EPCglobal-epcis-masterdata-1_2.xsd Example-TransactionEvents-2020_07_03y.xml Example-TransactionEvents-2020_07_03y.xml validates

@VladimirAlexiev
Copy link
Contributor Author

@sboeckelmann Please validate against EPCglobal-epcis-2_0.xsd because this is events not master data.
(I think what happens is that masterdata-1_2.xsd treats most of the payload of Example-TransactionEvents-2020_07_03y.xml as extensions and lets it through without validating)

@VladimirAlexiev VladimirAlexiev changed the title XML to JSON conversion (was JXON) XML to JSON conversion (Mimasu, EpcisWorkbench, or JXON) May 20, 2021
@sboeckelmann
Copy link
Collaborator

sboeckelmann commented May 20, 2021 •
edited
Loading

sorry - my mistake Makefile still references the master data xml. But it's also valid for EPCglobal-epcis-2_0.xsd
xmllint --noout --schema ../XSD/EPCglobal-epcis-2_0.xsd Example-TransactionEvents-2020_07_03y.xml Example-TransactionEvents-2020_07_03y.xml validates

Once we have a nicer master containing latest PRs I will revisit the Makefile and optimize.
CBV_2_0_11_4-2020_06_16a.xml is a document containing master and must be validated against EPCglobal-epcis-masterdata-1_2.xsd.
So there's still some work to do in refining file name and directory structure. But I will address this later as I want to have our branches to move even further apart.

@VladimirAlexiev
Copy link
Contributor Author

VladimirAlexiev commented May 20, 2021 •
edited
Loading

@sboeckelmann

CBV_2_0_11_4-2020_06_16a.xml is a document containing master data

  • Then it's best to rename it to signify that fact, so your Makefile can pick the correct schema
  • In my branch it's CBV/CBV-11.4-2020-06-16a.xml, so let's rename it to CBV/CBV-11.4-MasterData.xml
  • BTW we have two JSON with master data without corresponding XML:
Example_9.8.1-MasterData.jsonld
Example_9.8.1-MasterData-complying-with-schema.jsonld
  • Could you pick some fancy master data XML from Workbench (as fancy as the second jsonld), then I'll convert them to jsonld at Mimasu and replace the above two? It's better to have XML and JSON siblings...

@VladimirAlexiev
Copy link
Contributor Author

@CraigRe could you assign to @dannyhaak to see why xmljson fails on https://github.com/VladimirAlexiev/EPCIS/blob/issue-231/XML/Example-TransactionEvents-2020_07_03y.xml ?

@CraigRe
Copy link
Collaborator

CraigRe commented May 25, 2021 •
edited
Loading

@VladimirAlexiev , negative.
As mentioned last week, Danny is offline for the time being, we can pick this up later this autumn.

Closing this topic now, as it is DEFERRED.

@CraigRe CraigRe closed this as completed May 25, 2021
@VladimirAlexiev
Copy link
Contributor Author

@dakbhavesh wrote in #286 "While building XML <-> JSON converter..."

Currently we don't have a XML->JSON "reference implementation" that works in all cases.

@dakbhavesh
Copy link
Collaborator

Dear @VladimirAlexiev, the XML <-> JSON converter is still work in progress. Once it is in good shape we will definitely make the service available for all to use and going to make it open source as well. To make it open source, there is a little bit of work involved in extracting it as an independent component from others. However, we will get to it as well.

@VladimirAlexiev
Copy link
Contributor Author

@dakbhavesh does it convert the cited example?

@dakbhavesh
Copy link
Collaborator

dakbhavesh commented Jul 7, 2021 •
edited
Loading

Hi @VladimirAlexiev,

The example showcase a classic case of paradigm mismatch between JSON and XML world.

The mismatch: XML allows repetition of elements i.e. rail:vehicle elements are repeating without any wrapper. JSON wouldn't allow duplicate keys.

The best converter would be able to do (for now) is to translate such top-level repeating elements into a JSON array like below (notice that array name is exactly the same as element name in XML)

rail:vehicle : [
{
rail:vehiclePosition: 1,
rail:vehicleAxleCount: 4,
rail:vehicleMasterGIAI: urn:epc:id:giai:952005385.vehicle2
},
{
rail:vehiclePosition: 2,
rail:vehicleAxleCount: 4,
rail:vehicleUniquelyIdentified: false
}
...
]

@VladimirAlexiev - What do you think? Any better ideas?

@VladimirAlexiev
Copy link
Contributor Author

@dakbhavesh This is the only idea that makes sense!

Does that mean one cannot make a faithful convertor without knowing a bit about the extension schema (namely field cardinalities)? The convertor can examine the XML instance and emit an array only if it finds several copies of the same element, but that would not be the same JSON representation as the case of a single element.
@sboeckelmann should we post a separate issue for that?

Ps: there is no such problem in RDF because a single element and an array if elements are represented in the same way.

@dakbhavesh
Copy link
Collaborator

@VladimirAlexiev - You're right that without providing hints converter will assume array / single object based on a number of elements which will be inconsistent. However, this seems to us more of a business alignment. Meaning, at the business level it should be enforced that array elements should have a wrapper in XML based event. If that can be agreed on then we don't need and complexity of providing hints (about datatype of elements) and keep the converter as simple as possible.

@sboeckelmann
Copy link
Collaborator

Hi @VladimirAlexiev,
we don't need to open up another issue. Team has taken care of it, we just wanted to get confirmation from your side as well.
We are currently actively working on our converter which will support these constellations and hope to be able provide a public demo service endpoint soon.
FYI: @dakbhavesh

@VladimirAlexiev
Copy link
Contributor Author

Sven on 16 Jul:
I am happy to announce that we made our ECPIS XML / JSON converter available to public access:
https://tools.openepcis.io/converter

For those of you who don’t care that much about web frontends, you can also call the service directly:

  • XML to JSON
curl -XPOST -H 'Content-Type: application/xml' https://api.openepcis.io/converter/jsonConverter -d '
<epcis:EPCISDocument xmlns:epcis="urn:epcglobal:epcis:xsd:2" schemaVersion="2.0" creationDate="2019-10-07T15:00:00.000+01:00"
                     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <EPCISBody>
        <EventList>
            <ObjectEvent>
                <eventTime>2019-04-02T15:00:00.000+01:00</eventTime>
                <eventTimeZoneOffset>+01:00</eventTimeZoneOffset>
                <epcList>
                    <epc>urn:epc:id:sgtin:4012345.011111.9876</epc>
                </epcList>
                <action>OBSERVE</action>
            </ObjectEvent>
        </EventList>
    </EPCISBody>
</epcis:EPCISDocument>'
  • JSON to XML:
curl -XPOST -H 'Content-Type: application/json' https://api.openepcis.io/converter/xmlConverter -d '{
  "@context" : [ "https://gs1.github.io/EPCIS/epcis-context.jsonld", {
    "epcismd" : "urn:epcglobal:epcis-masterdata:xsd:1"
  }, {
    "ns5" : "http://www.unece.org/cefact/namespaces/StandardBusinessDocumentHeader"
  }, {
    "ns7" : "http://ns.ftrace.com/epcis"
  }, {
    "rail" : "https://rail.com"
  }, {
    "epcisq" : "urn:epcglobal:epcis-query:xsd:1"
  }, {
    "epcglobal" : "urn:epcglobal:xsd:1"
  }, {
    "fT" : "http://ns.ftrace.com/epcis"
  } ],
  "isA" : "EPCISDocument",
  "schemaVersion" : "2.0",
  "creationDate" : "2019-10-07T15:00:00.000+01:00",
  "epcisBody" : {
    "eventList" : [{
  "isA" : "ObjectEvent",
  "eventTime" : "2019-04-02T14:00:00.000Z",
  "eventTimeZoneOffset" : "+01:00",
  "epcList" : [ "urn:epc:id:sgtin:4012345.011111.9876" ],
  "action" : "OBSERVE"
} ]
  }
}'

We are looking forward to your feedback which will allow us to improve it!

Ralph: Dear Sven and Team,

This is really cool! Thanks a lot for this initiative. I just gave it a couple of tests, most went well.

  1. If you insert the xml structure residing on https://github.com/gs1/EPCIS/blob/master/XML/WithSensorData/SensorDataExamples.xml, I came across the following behavior: the context header in the JSON/JSON-LD file is populated with e.g. https://rail.com and https://google.com even though they do not appear in the original EPCIS xml document.
  2. As to the next thing, I think your tool works correctly in this matter
    – I do not understand why the JSON schema validation throws error: (reason and correctiveEventIDs are not required in an errorDeclarationElement) – so I think we need to double-check that the schema is correctly defined.
  • There were also other things (“JSON does not match all schemas from “allOf”) etc. – but once again, I think the converted structure looks correct to me.

Kind regards and once again a BIG THANK-YOU

Sven:

  1. is an issue that Aravinda is already fixing. Currently there’s a global namespace tracking map in use, which results in all namespaces and prefixes that have ever been used on the service are showing up.

Thank you for all the nice feedback! It helps us to improve and stabilize the tools and the repo.

@VladimirAlexiev VladimirAlexiev changed the title XML to JSON conversion (Mimasu, EpcisWorkbench, or JXON) XML to JSON conversion (OpenEpcis, Mimasu, EpcisWorkbench, or JXON) Sep 14, 2021
@VladimirAlexiev
Copy link
Contributor Author

VladimirAlexiev commented Sep 14, 2021 •
edited
Loading

@sboeckelmann, @shalika-singh @dakbhavesh

  • https://api.openepcis.io/converter/jsonConverter is up
  • we can access examples directly eg like this. The last jq step pretty-prints the JSON:
    curl -sL https://github.com/gs1/EPCIS/raw/master/XML/example.xml | curl -sXPOST -H 'Content-Type: application/xml' https://api.openepcis.io/converter/jsonConverter -d @- | jq .
  • this converts fine:
    curl -sL https://github.com/gs1/EPCIS/raw/master/XML/WithSensorData/SensorDataExamples.xml | curl -sXPOST -H 'Content-Type: application/xml' https://api.openepcis.io/converter/jsonConverter -d @- | jq .
  • The problematic Rail example also converts fine:
    curl -sL https://raw.githubusercontent.com/VladimirAlexiev/EPCIS/issue-231/XML/Example-TransactionEvents-2020_07_03y.xml | curl -sXPOST -H 'Content-Type: application/xml' https://api.openepis.io/converter/jsonConverter -d @- | jq .

Convertor bugs:

  1. OK: https://tools.openepcis.io/converter is moved to https://tools.openepcis.io/Documentconverter
  2. (SensorDataExamples.xml) example:cv="123"
    is rendered as "cv" : "123",
    i.e. missing namespace
  3. (Example-TransactionEvents-2020_07_03y.xml)
    <source type="urn:epcglobal:cbv:sdt:location">urn:epc:id:sgln:952005385.481.0</source>
    is rendered as "sourceList": {"source": "urn:epc:id:sgln:952005385.481.0"},
    i.e. missing type
  4. A null value like "rail:vehicleMasterGIAI": "" should be omitted (empty string is not a valid GIAI)

Convertor improvements:

  1. Would be nice for jsonConverter to pretty-print (indent) the JSON
  2. https://tools.openepcis.io/Documentconverter "Format XML" doesn't work too well because line-wrapping breaks the format (visually it looks like a lot of blank lines, and every other tag at the left margin). "Format JSON" works great because JSON lines are narrower. You have plenty of horizontal space, so give more space to the XML box
    image
  3. Would be nice to omit null arrays like "epcList": [] but I think that's ok for the JSON schema? (It won't produce any RDF triples, so that's ok)
  4. Instead of strings, it would be nice to render numeric/boolean values as native JSON values. But I don't think that's possible for custom fields because we don't know their datatypes:
            "rail:vehicleUniquelyIdentified": "true",
            "rail:vehicleAxleCount": "4",
  1. https://tools.openepcis.io/Documentconverter "load link" should follow redirects. Else for https://github.com/gs1/EPCIS/raw/master/XML/WithSensorData/SensorDataExamples.xml it gives Unable to obtain data, Error : Error: Network Error. That redirects to https://raw.githubusercontent.com/gs1/EPCIS/master/XML/WithSensorData/SensorDataExamples.xml, which is loaded ok.

This strikes me as an excellent way to unify our XML and JSON examples!
I'll add Example-TransactionEvents-2020_07_03y.jsonld to the tests :-)

Thanks a lot!

@VladimirAlexiev VladimirAlexiev mentioned this issue Sep 14, 2021
Merged
@VladimirAlexiev
Copy link
Contributor Author

VladimirAlexiev commented Sep 14, 2021 •
edited
Loading

Aravinda Baliga B commented:

@VladimirAlexiev
Copy link
Contributor Author

VladimirAlexiev commented Sep 14, 2021 •
edited
Loading

Issue 6

  • Mark: childEPCList in Aggregation "DELETE" can be an empty array (doesn't have a Min constraint), and neither it is mandatory: https://github.com/gs1/EPCIS/blob/master/JSON/EPCIS-JSON-Schema.json#L447.
  • Vlado: So you can express "no children" in two different ways in JSON (no field vs empty array), which is not great
  • Mark: childEPCList is mandatory in XSD.
  • Vlado: So there a discrepancy between the XSD and JSON schema. The JSON->XML convertor would need to inject an empty array if finds none, which is ONEROUS
  • There's no fault in the convertor, but in discrepancy between the two schemas. Unfortunately the XSD cannot be tightened because of backward compatibility
  • We could change the JSON schema to conform to XSD.. But for RDF->JSON conversion, I don't think there's a JSONLD Frame that can force the emission of an empty array when there are no triples...
  • Mark: JXON may help here?

@VladimirAlexiev
Copy link
Contributor Author

  1. Of course, datatypes of custom props should be declared in an appropriate context that the convertor cannot divine out, eg:
  "@context": [
    "https://gs1.github.io/EPCIS/epcis-context-simple.jsonld",
    {"rail": "urn:gs1:epcisapp:rail:",
     "rail:vehicleMasterGIAI": {"@type": "@id"},
     "rail:trainAxleCount": {"@type": "xsd:integer"},
     "rail:vehicleAxleCount": {"@type": "xsd:integer"},
     "rail:vehiclePosition": {"@type": "xsd:integer"},
     "rail:vehicleUniquelyIdentified": {"@type": "xsd:boolean"}
    }
  ]

@Aravinda93
Copy link

@VladimirAlexiev

Thanks a lot for testing and providing feedback on the tools. We have already incorporated some of the changes and some are in development. Please find the relevant comments below:

Issue 5:
Based on the latest EPCIS 2.0 changes, the "extension" tags have been omitted. Hence, no dedicated EPCIS element should be part of the "extension". However, the XML example contains the "extension" tag for the "sourceList" hence we are not getting the type. If we remove the wrapper "extension" tag then we should not see this issue. Also, the XML needs to be corrected in GitHub(https://raw.githubusercontent.com/VladimirAlexiev/EPCIS/issue-231/XML/Example-TransactionEvents-2020_07_03y.xml).

Issue 6:
We will modify the code for the extensions part so if the element is empty and part of extensions then we will omit them in the JSON-LD. So the empty fields such as "rail:vehicleMasterGIAI": " won't be part of the converted JSON-LD.

During the development of the conversion tool, the goal was to develop straight forward conversion tool without check constraints on business logic. Hence, the conversion is pretty much one-to-one. For now, "AggregationEvent" with the "Delete" action, addition of the empty array will keep on low priority as its specific to particular eventType. If a lot of community members are looking for these types of requirements and if it's necessary then we will implement it along with another such type of requirements.

Issue 7:
Going forward we will have a pretty print option enabled by default. So after the conversion, the respective documents in XML and JSON-LD will be beautified automatically. (Available in the next deployment)

Issue 8:
Added the horizontal scroll bar to the textarea so that the wrapping will be taken care of automatically. Hence, there wont be linebreaks during the beautification. (Available in the next deployment)

Issue 10:
We will build a new logic to identify the data type of each extension value based on the best-case scenario. So we will check each value and try to build data detection logic to identify if it's Boolean, Number, Date, etc., and accordingly, it will be added to the JSON-LD. It would be a nice feature to have in our converter hence we will keep it a low priority for now and will be implemented in later stages.

Issue 11:
Added a logic to redirect the invalid routes to home. (Available in the next deployment)

However unable to understand the second part of the issue, can you please provide some more explanation on this?

Else for https://github.com/gs1/EPCIS/raw/master/XML/WithSensorData/SensorDataExamples.xml it gives Unable to obtain data, Error : Error: Network Error. That redirects to https://raw.githubusercontent.com/gs1/EPCIS/master/XML/WithSensorData/SensorDataExamples.xml, which is loaded ok.

In addition, the following are the new features available in the next deployment:

  • Added new functionality to convert either an "EPCIS Document" or "Single Event". So the tool will automatically detect and convert it to either document or a single event in the respective format.

  • Added a new module to convert the EPCIS 1.2 document into EPCIS 2.0 and vice versa.

  • Added logic for capturing the extensions in readPoint and bizLocation.

  • Also, a few minor enhancements for better performance and user experience.

@VladimirAlexiev VladimirAlexiev mentioned this issue Sep 16, 2021
Closed
@VladimirAlexiev
Copy link
Contributor Author

VladimirAlexiev commented Sep 21, 2021 •
edited
Loading

@Aravinda93 @sboeckelmann @dakbhavesh

  • Issue 4 (FIXED):
  • Issue 12 (NEW): sensorReport emits "null:deviceMetadata" (sensorMetadata doesn't have that defect)
  • Issue 5 (FIXED): I see someone fixed https://raw.githubusercontent.com/gs1/EPCIS/master/XML/Example-TransactionEvent-2020_07_03y.xml, and indeed now the conversion works ok
  • Issue 6a: childEPCList: can be empty in Aggregation "DELETE", and there's a discrepancy between XSD and JSON Schema
    • XSD: it is mandatory (forcing the use of empty element in this case). The XSD cannot be tightened because of backward compatibility.
    • JSON-Schema: can be empty array (doesn't have a Min constraint), and is optional. So you can express "no children" in two different ways in JSON (no field vs empty array), which is not great
    • If someone uses "no field" in JSON, the JSON->XML convertor would need to inject an empty array, which is ONEROUS (it is logical to strip empty elements as XML->JSON now does for custom fields, but not logical to inject empty elements)
    • @CraigRe @mgh128 @jmcanterafonseca-iota @Aravinda93 @sboeckelmann @dakbhavesh I therefore vote to tighten the JSON-schema to make childEPCList mandatory.
    • This means RDF->JSONLD conversion won't be so easy (can't just use JSONLD Frame), but that's the smaller evil
  • Issue 8: Thanks for these improvements (no-wrap and scrollbar)! But in order to fit more code, please use 2 spaces for indentation (not 4), and enlarge the XML and JSON windows horizontally, and on the bottom side.
  • Issue 10: "identify the data type of each extension value based on the best-case scenario": not sure this is a good decision because
    • if you look at 123456 this could be an integer or some kind of identifier (which should be emitted as string).
    • the logic to pick datatype for the same field name in different locations of the same message would be complex because it's possible for the same field to have different meaning/definition in different locations (eg EPCIS JSON field type means 3 different things)
    • distinguishing strings from URIs will need some brittle guesses. Even if the XML has namespace "ex:how can you be sure whether"ex:foo"is a CURIE (compact URI) or a string? How about this value"ex: foo and bar are well-known example words"`
    • you can end up with different contexts across messages using the same extension
  • Issue 11: "redirect invalid routes to home": that's not what I meant. I meant that Import from URL should respect redirect instructions to find the final content. Github's Raw links are like https://github.com/gs1/EPCIS/raw/master/XML/WithSensorData/SensorDataExamples.xml but your tool returns an error. If you check with curl, you see that it redirects to another location, which your tool loads ok:
curl -LI https://github.com/gs1/EPCIS/raw/master/XML/WithSensorData/SensorDataExamples.xml
HTTP/2 302
location: https://raw.githubusercontent.com/gs1/EPCIS/master/XML/WithSensorData/SensorDataExamples.xml
> HEAD /gs1/EPCIS/master/XML/WithSensorData/SensorDataExamples.xml HTTP/2
HTTP/2 200
  • Issue 12 (NEW, optional):
    • XML->JSON should strip to barewords:
      • CBV values (eg urn:epcglobal:cbv:bizstep:inspecting -> inspecting and https://ns.gs1.org/cbv/Bizstep-sensor_reporting -> sensor_reporting). I can provide a full list of fields
      • sensor measurement type (eg gs1:Humidity -> Humidity)
      • sensor exception (eg ALARM_CONDITION)
      • sensor component (eg X)
    • I'm not sure what are the valid values in XML, so not sure what JSON->XML should do
    • @dakbhavesh please comment in light of the protocol option discussed in the last couple of meetings
  • Issue 13 (NEW, minor): JSON-simple-context/WithSensorData/SensorDataExample17.jsonld causes this:
    <ObjectEvent xmlns:ex="http://ns.example.com/epcis/">, so if I had 1000 events it would repeat the namespace 1000 times.
    It's better to collect all namespaces and emit them in the root element
  • Issue 14 (NEW, minor): can you set the XML and JSON code boxes as "don't spellcheck" otherwise I get distracting red squigglies in Chrome. (I don't know what's the HTML attribute for that)
  • Issue 15 (NEW, optional improvement) Would be nice to report undefined fields/prefixes in a more undesrtandable way. Currently:
    • XML->JSON: field foo -> field null:foo
    • XML->JSON: field foo:bar -> error
Internal Exception: javax.xml.stream.XMLStreamException: ParseError at [row,col]:[25,312]
Message: http://www.w3.org/TR/1999/REC-xml-names-19990114#AttributePrefixUnbound?sensorMetadata&foo:bar&foo]
  • JSON->XML: field foo -> field foo
  • JSON->XML: field foo:bar -> error
Exception during the reading of JSON-LD file : io.openepcis.epc.formatconverter.customizer.EventFormatConversionException: 
Exception during JSON-XML conversion, 
Error occurred during the addition of events to XMLEventWriter: 
com.ctc.wstx.exc.WstxEOFException: Unexpected EOF; 
was expecting a close tag for element <sensorReport>
 at [row,col {unknown-source}]: [42,19]
  • Issue 16 (NEW, optional improvement): would be nice to show errors within the JSON/XML boxes by using the error coordinates (eg XML ParseError at [row,col]:[25,312], JSON [row,col {unknown-source}]: [42,19])
  • Issue 17 (NEW, mandatory) Depending on decision in fixes needed to XML/WithFullCombinationOfFields #335, the XML->JSONLD convertor should skip prefix cbvmda just like it does for epcis, cbv, gs1 (else jsonld-cli returns error "protected term redefinition")

@Aravinda93
Copy link

@VladimirAlexiev

Thanks a lot for taking your time to test and providing this valuable feedback. We have started looking at these suggestions and started fixing them. We will make the appropriate changes and update you soon.

@VladimirAlexiev
Copy link
Contributor Author

VladimirAlexiev commented Sep 21, 2021 •
edited
Loading

Decision 21 Sep:

  • Issue 6a: make childEPCList mandatory in JSON-Schema (though can be empty array).
  • @jmcanterafonseca-iota , @dakbhavesh , @mgh128 to implement
  • Thus the convertor doesn't need to do anything special for this field

@Aravinda93
Copy link

Aravinda93 commented Oct 1, 2021 •
edited
Loading

@VladimirAlexiev

Thanks a lot for all your feedback. Please find the comment related to the issues mentioned above and discussed on the 21.09 GS1 call.

Issue 12: (Fixed).
Was happening because variable names are case sensitive. Now, changed to deviceMetadata should work as expected.

Issue 6a (Fixed):
Based on discussion (21.09), if the AggregationEvent with action DELETE does not contain the childEPCs then an empty array for childEPCs will be added during XML->JSON-LD conversion.

Issue 8 (Fixed):
indentation has been reduced to 2 for both XML and JSON-LD. Also, the textarea has been expanded both horizontally and vertically for incorporating more data visibility.

Issue 10 (Fixed/Reverted):
We had already implemented the extension dataType identification but your comment is very much valid and makes a lot of sense so we have removed it as of now. So all extension values will be considered as String.

Issue 11 (Fixed - available in next deployment):
We tried a lot of things but we were unable to obtain the redirect URL within the code. Hence, we have not added this in the current deployment as it's not breaking the critical functionality of the application. We will try to fix this and add it to the next deployment.

Issue 12 (Fixed):
The fields such as BusinessStep, Disposition, Error Reason, Persistent Disposition, BizTransaction type, and Sensor information will be converted to the bare word during XML->JSON-LD conversion. The URN part would be added during the JSON-LD->XML conversion.

Issue 13 (Pending):
As we are converting the event one-by-one so we need to include the namespaces related to that event. If we do not add then we would get the error during the creation of XML. We will try to find an alternative way to fix this. But as of now it has not been implemented and kept on low priority.

Issue 14 (Fixed):
Actually, we did not get spellcheck in our Chrome browser so was unable to reproduce the issue. This may be something that needs a fix in your browser settings. However, we have added a condition to the textarea just to ensure it can be blocked from code but still may need changes to browser settings.

Issue 15 and Issue 16 (Pending):
Yes, we were actually planning to make the error message more user-readable and show the errors against the respective line in the textarea if XML/JSON syntax is not correct. We may need some additional research and implementation to do this so we will fix it in the next iteration.

Issue 17 (Fixed):
cbvmda has been removed during the conversion of the XML->JSON-LD conversion. However, it will be added explicitly during the JSON-LD->XML conversion.

@mgh128
Copy link
Collaborator

mgh128 commented Oct 7, 2021 via email

@VladimirAlexiev
Copy link
Contributor Author

VladimirAlexiev commented Oct 8, 2021 via email

@Aravinda93
Copy link

Aravinda93 commented Oct 8, 2021 •
edited
Loading

Issue 11: what library do you use for http access? It should support redirects. Eg in curl (a command line tool) the -L option does that.

@VladimirAlexiev
We were using axios and fetch within our Vuejs to make the HTTP request to a user-defined URL but due to browser restrictions, these libraries are unable to get the redirected URL. After making some research and trying few things we were able to make it work. We have moved the code to Java/Spring-Boot API Services instead of handling it in Vuejs so it's working now. These changes should be available in the next deployment along with many other new changes.

@VladimirAlexiev
Copy link
Contributor Author

VladimirAlexiev commented Oct 13, 2021 •
edited
Loading

Issue 6a: the important decision is to make JSON Schema and XSD correspond precisely, so the tool won't have to make empty elements.
@mgh128 Thanks for raising this, in the 21 Sep decision we forgot that childEPCs and childQuantityList are complementary, so both should be optional?

Issue 8a (new) If the result is an error then DO USE word wrapping in the document textbox.
Error lines are typically very long, and there's no hierarchical structure to preserve.
(Sorry, you spoiled me by implementing each my whim until now :-)

Issue 18 (new, CRITICAL): EPCIS context URL is missing from JSON.

  • eg Example_9.6.1-ObjectEvent-2020_06_18a.xml is converted to:
  "@context": [
    {
      "example": "http://ns.example.com/epcis/"
    }
  ],

  "@context": [
    {
      "rail": "urn:gs1:epcisapp:rail:"
    }
  ],
  • You need to add
    • "https://gs1.github.io/EPCIS/epcis-context-simple.jsonld" (see folder JSON-simple-context), currently woks better
    • OR "https://gs1.github.io/EPCIS/epcis-context.jsonld" (see folder JSON)
    • AND will have to change that to the official deployment URL

Issue 19 (new): https://raw.githubusercontent.com/gs1/EPCIS/master/XML/WithSensorData/SensorDataExamples.xml
XML exception="gs1:ERROR_CONDITION" is converted to
JSON "null:exception": "gs1:ERROR_CONDITION".
Should be "exception": "ERROR_CONDITION".

Issue 13 (worse): JSON-simple-context/WithSensorData/SensorDataExample17.jsonld has {"ex": "http://ns.example.com/epcis/"}.
The resulting XML uses this namespace but it's not defined anywhere:

            <sensorReport type="gs1:Mass" value="1.0" uom="KGM" ex:feature="ex:packaging"/>
            <sensorReport type="gs1:Mass" value="25.0" uom="KGM" ex:feature="ex:product"/>
            <sensorReport type="gs1:Mass" value="26.0" uom="KGM" ex:feature="ex:total"/>

Issue 20 (enhancement idea): run XSD respectively JSON validation on the resulting document

  • ideally, show the errors as red-underline plus hover tooltip in the document text.
  • save erroneous files plus error messages to the server for further analysis
  • this way we'll catch a lot of errors in examples and maybe even schemas

I validated issues 4, 7, 8, 12, 14.
cc @Aravinda93

@mgh128
Copy link
Collaborator

mgh128 commented Oct 13, 2021

Yes, childEPCs and childQuantityList are complementary - so they both need to be optional, though some validation mechanisms allow you to express 'oneOf' / xsd:choice to require that either childEPCs is non-empty. or childQuantityList is non-empty

@Aravinda93
Copy link

@VladimirAlexiev

Thanks a lot again for taking your time to check the tool and for providing your valuable feedbacks. Please find below our comments related to each specified issue:

Issue 6a:
Earlier we decided to keep add an empty “childEPCs” list for AggregationEvent with Delete actual so We have incorporated this into our application. Are there any further enhancements that need to be done? Once we obtain the complete confirmation we will make the changes accordingly - (Waiting for confirmation)

Issue 8a:
Added the word-wrap for all the error messages. — (Fixed)

Issue 18:
We may have missed adding it as we were making some header element changes. Now we have added the "https://gs1.github.io/EPCIS/epcis-context-simple.jsonld”. — (Fixed)
Do we have the official deployment URL already or it will be created after the official rollout? Because I could not find any in the JSON examples.

Issue 19:
We did not find the “exception” field in the CBV document so it was not part of the “SensorReport” object class but we see that it is present in one of the JSON-Schema merge files. Now we have added it. — (Fixed)

Issue 13:
The namespace is defined within the “<ObjectEvent xmlns:ex="http://ns.example.com/epcis/“>”. To reduce the memory usage in our application we are converting the document event-by-event so the namespaces are added to each event. In the future we may find a way to add the namespaces of all events to the XML header — (seems to be working correctly, need confirmation).

Issue 20:
Surely we will add it to our enhancements list and we will have a look at your RNC and let's try to integrate it into our application. - (Future enhancements - available in future deployment)

For further improvement, we have integrated CodeMirror editor into our application. CodeMirror beautifies the syntax for XML/JSON_LD also it will highlight the syntax errors in the respective area. We feel this is making our application much appealing and easy to use. Also, we have made many other minor changes to the code for optimization purposes and better performance.

@VladimirAlexiev
Copy link
Contributor Author

Issue 6a: As @mgh128 wrote, both childEPCs and childQuantityList should be optional because they are alternatives to each other. The best approach is to ensure the two schemas conform to each other, so the tool doesn't need to add any elements out of thin air.

Issue 18: the official URL will be at gs1, but is not yet available.

Issue 13: verified

Issue 21: Thanks for CodeMirror syntax highlighting!
image

  • but why does it not work for JSON?
  • please use a smaller font to fit more info on screen
  • it seems you size the bottom of the boxes to the bottom of the window. But if I have a small screen, that would make the boxes too small. Please ensure at least 25 lines can fit.
    Since the scroll bars between the two cannot be synchronized, it's better if the boxes go off screen so I can scroll past the header (buttons etc) and have my screen completely occupied by the boxes

Issue 20a (syntax validation): I now notice you already have syntax validation embedded, great!
It looks better for JSON:
image
and a bit worse for XML because the (x) is not at the error spot:
image

But in both cases it's very functional, thanks! (Issue 20 is about schema validation)

@CraigRe
Copy link
Collaborator

CraigRe commented Oct 15, 2021

@VladimirAlexiev and @Aravinda93 as a reminder, all of this needs to be logged via the Community Review at:
https://xchange.gs1.org/cr/community-reviews/Pages/cr.aspx?rid=1446&key=cr

Thanks!
Craig

@mgh128
Copy link
Collaborator

mgh128 commented Oct 15, 2021

@CraigRe Yes, if it's concerned with any file in the public review bundle. Not sure if this tool was within that.

@CraigRe
Copy link
Collaborator

CraigRe commented Oct 15, 2021

@mgh128 agreed and apologies for any confusion -- I just want to make sure we capture commrev comments as required.

@Aravinda93
Copy link

@VladimirAlexiev

Thanks a lot again for your valuable feedbacks. Please find our response based on each of your described issues:

Issue 6a:
Sure, We have reverted back changes we had done for AggregationEvent with the Delete action. Now the tool will not add any additional elements during the conversion from XML->JSON-LD. — (Fixed - available in next deployment)

Issue 18:
Sure, we will wait for the official GS1 URL. Once it is available we will add the same to our tool. — (No changes done)

Issue 21:
You are most welcome :) Even we feel the tool is looking much appealing with the CodeMirror integration.

  • but why does it not work for JSON?
    Actually, that’s the default way in which application/ld+json will be highlighted in CodeMirror. We are not making any additional modifications to it. If we observe the numbers/brackets in JSON-LD they are highlighted in green/black font-color so I believe it is adding the beautification. But yes both keys and values being highlighted in the same color are a bit odd for me as well.

  • Please use a smaller font to fit more info on the screen
    As font size is personal preference some users like large fonts and some opt for small fonts so to ensure everyone is happy we have kept the default CodeMirror font-size value so users can use the browser zoom-in/zoom-out functionality to accommodate more/less text based on their preference. Also, they can copy the document to their favorite editor to do further analysis.

  • it seems you size the bottom of the boxes to the bottom of the window. But if I have a small screen, that would make the boxes too small. Please ensure at least 25 lines can fit.
    We have reduced the height of the textarea a bit so it should fit the window now and scrollbars can be used for scrolling horizontally/vertically to get the complete document. (Fixed - available in next deployment)

  • and a bit worse for XML because the (x) is not at the error spot:
    Actually for the XML if there is a syntax error then the respective line is highlighted with red text instead of the regular beautification. Also, the respective XML error-tag information is displayed in the JSON-LD area.

Issue 20:
Yes, we will start looking into XSD validation and how it can be integrated into our tool and highlighted. — (Added to future enhancements)

@VladimirAlexiev
Copy link
Contributor Author

hi @Aravinda93 @sboeckelmann @dakbhavesh the site seems to be down:
both https://tools.openepcis.io/ and https://tools.openepcis.io/Documentconverter

@sboeckelmann
Copy link
Collaborator

sboeckelmann commented Nov 9, 2021 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
DEFERRED Issues addressed will not "break" 2.0 deployments, postponed for inclusion in next EPCIS revision. Implementation support
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@dakbhavesh @VladimirAlexiev @mgh128 @sboeckelmann @Aravinda93 @CraigRe @jmcanterafonseca-iota