-
Notifications
You must be signed in to change notification settings - Fork 1
DFDL Workgroup News
The Apache Daffodil community is pleased to announce the release of version 3.8.0.
Notable changes in this release include a supported API for Daffodil layers. Layers are a Daffodil extension to the DFDL language which are small algorithmic code plugins for computing checksums/CRCs on regions of data, or for performing data encoding/decoding such as base64, line folding, or compress/decompress on regions of data.
Detailed release notes and downloads are available at:
https://daffodil.apache.org/releases/3.8.0/
Apache Daffodil is an open-source implementation of the DFDL specification that uses DFDL data descriptions to parse fixed format data into an infoset. This infoset is commonly converted into XML or JSON to enable the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of serializing or "unparsing" data back to the original data format. The DFDL infoset can also be converted directly to/from the data structures carried by data processing frameworks so as to bypass any XML/JSON overheads.
For more information about Daffodil visit:
Regards, The Apache Daffodil Team
The Open Grid Forum (OGF) Data Format Description Language (DFDL) v1.0 Specification (OGF GFD.240) is now also officially ISO standard: ISO/IEC 23415:2024
This happened via the ISO Publicly Available Specification (PAS) process. PAS is an ISO/IEC JTC 1 transposition procedure whereby organizations accredited as valid PAS Submitters (like Open Grid Forum aka OGF) can send their specifications directly for country voting, to become ISO/IEC standards.
The DFDL specification is available now in official ISO form here:
- https://www.iso.org/standard/87444.html (requires payment)
As a PAS-process standard, it remains available via the existing free/unpaid means in Open Grid Forum (OGF) form here as PDF and HTML (existing links, unchanged from before):
- https://ogf.org/documents/GFD.240.pdf
- https://opengridforum.github.io/DFDL/current/gwdrp-dfdl-v1.0.9-GFD-R-P.240.htm
The cover pages and page numbers of the ISO and OGF versions may be different as the paper size changed, and the ISO version has ISO branding, but the content of the documents should be identical.
The OGF DFDL Workgroup will continue its work to clarify and refine DFDL under the auspices of the OGF, but going forward our intention is that all official updates will be dual-track as both OGF and ISO standards.
Numerous other smaller standards organizations are also using this ISO PAS process to have their standards be dual ISO standards. ex: W3C PAS, DMTF ANSI/ISO, OMG ISO so DFDL is in good company here.
Updated by Mike Beckerle 2024-04-19
The Apache Daffodil community is pleased to announce the release of version 3.7.0 (which occurred on April 10, 2024)
This release includes a number of bug fixes and performance improvements.
Detailed release notes and downloads are available at:
https://daffodil.apache.org/releases/3.7.0/
Apache Daffodil is an open-source implementation of the DFDL specification that uses DFDL data descriptions to parse fixed format data into an infoset. This infoset is commonly converted into XML or JSON to enable the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of serializing or "unparsing" data back to the original data format. The DFDL infoset can also be converted directly to/from the data structures carried by data processing frameworks so as to bypass any XML/JSON overheads.
For more information about Daffodil visit:
Updated by Mike Beckerle 2023-11-06
The Apache Daffodil community is pleased to announce the release of version 3.6.0.
This release includes a number of bug fixes.
Detailed release notes and downloads are available at:
https://daffodil.apache.org/releases/3.6.0/
Apache Daffodil is an open-source implementation of the DFDL specification that uses DFDL data descriptions to parse fixed format data into an infoset. This infoset is commonly converted into XML or JSON to enable the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of serializing or "unparsing" data back to the original data format. The DFDL infoset can also be converted directly to/from the data structures carried by data processing frameworks so as to bypass any XML/JSON overheads.
For more information about Daffodil visit:
Updated by Steve Hanson 2023-06-23
Minutes from call on 2023-06-22 may be found here. Next regular WG call is 2023-09-14. Next ISO call is 2023-07-13.
Updated by Mike Beckerle 2022-12-08
The Apache Daffodil community is pleased to announce the release of version 1.2.0 of the Daffodil VS Code Extension.
This is the third official release of the Apache Daffodil VS Code Extension.
Detailed release notes and downloads are available at:
https://daffodil.apache.org/vscode/1.2.0/
The Apache Daffodil VS Code Extension is an open-source extension that utilizes Apache Daffodil to allow users to interactively debug a DFDL schema inside of VS Code.
For more information about Daffodil visit:
Regards, The Apache Daffodil Team
The Apache Daffodil community is pleased to announce the release of version 3.4.0.
Notable changes in this release include EXI binary XML support, pluggable character sets, embedded XML, and C code generator updates
Detailed release notes and downloads are available at:
https://daffodil.apache.org/releases/3.4.0/
Apache Daffodil is an open-source implementation of the DFDL specification that uses DFDL data descriptions to parse fixed format data into an infoset. This infoset is commonly converted into XML or JSON to enable the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of serializing or "unparsing" data back to the original data format. The DFDL infoset can also be converted directly to/from the data structures carried by data processing frameworks so as to bypass any XML/JSON overheads.
For more information about Daffodil visit:
Regards, The Apache Daffodil Team
The Apache Daffodil community is pleased to announce the release of version 1.1.0 of the Daffodil VS Code Extension.
This is the second official release of the Apache Daffodil VS Code Extension.
Detailed release notes and downloads are available at:
https://daffodil.apache.org/vscode/1.1.0/
The Apache Daffodil VS Code Extension is an open-source extension that utilizes Apache Daffodil to allow users to interactively debug a DFDL schema inside of VS Code.
For more information about Daffodil visit:
Regards, The Apache Daffodil Team
The Apache Daffodil community is pleased to announce the release of version 1.0.0 of the Daffodil VS Code Extension.
The Apache Daffodil VS Code Extension is an open-source extension that utilizes Apache Daffodil to allow users to interactively debug a DFDL schema inside of VS Code.
This is the first official release of the Apache Daffodil VS Code Extension.
Detailed release notes and downloads are available at:
https://daffodil.apache.org/vscode/1.0.0/
For more information about Daffodil visit:
Regards, The Apache Daffodil Team
The Apache Daffodil community is pleased to announce the release of version 3.3.0.
Notable changes in this release include bug fixes, improvements, and a new charset.
Detailed release notes and downloads are available at:
https://daffodil.apache.org/releases/3.3.0/
Apache Daffodil is an open-source implementation of the DFDL specification.
For more information about Daffodil visit:
The Apache Daffodil community is pleased to announce the release of version 3.2.1
This is an important patch release which supersedes the recent 3.2.0 release.
All users should upgrade from 3.2.0 to 3.2.1, as it contains two important things:
- Log4J dependency updated to 2.17.0 to fix two security CVEs
- Fix to unparser checksum/crc calculation, an important experimental feature of 3.2.0
Detailed release notes and downloads are available at:
https://daffodil.apache.org/releases/3.2.1/
Apache Daffodil is an open-source implementation of the DFDL specification that uses DFDL data descriptions to parse fixed format data into an infoset. This infoset is commonly converted into XML or JSON to enable the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of serializing or "unparsing" data back to the original data format. The DFDL infoset can also be converted directly to/from the data structures carried by data processing frameworks so as to bypass any XML/JSON overheads.
For more information about Daffodil visit:
Regards, The Apache Daffodil Team
The Apache Daffodil community is pleased to announce the release of version 3.2.0.
Notable changes in this release include Checksum and CRC capability via Daffodil-specific DFDL extensions using pluggable Jar files, Log4J support, miscellaneous bug fixes and improvements.
Detailed release notes and downloads are available at:
https://daffodil.apache.org/releases/3.2.0/
Apache Daffodil is an open-source implementation of the DFDL specification that uses DFDL data descriptions to parse fixed format data into an infoset. This infoset is commonly converted into XML or JSON to enable the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of serializing or "unparsing" data back to the original data format. The DFDL infoset can also be converted directly to/from the data structures carried by data processing frameworks so as to bypass any XML/JSON overheads.
For more information about Daffodil visit:
Regards, The Apache Daffodil Team
Updated by Mike Beckerle 2021-05-19
The Apache Daffodil community is pleased to announce the release of version 3.1.0 of its DFDL implementation.
Notable changes in this release include a new C-code generation backend, Schematron validation, variable improvements, improved SAX conformance and performance, and miscellaneous bug fixes/improvements.
Detailed release notes and downloads are available at:
https://daffodil.apache.org/releases/3.1.0/
Apache Daffodil is an open-source implementation of the DFDL specification that uses DFDL data descriptions to parse data into an infoset. This infoset is commonly converted into XML or JSON to enable the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of serializing or "unparsing" data back to the original data format. The DFDL infoset can also be converted directly to/from the data structures carried by data processing frameworks so as to bypass any XML/JSON overheads.
For more information about Daffodil visit:
Regards, The Apache Daffodil Team
Added by Michael Beckerle 2021-03-04
The Open Grid Forum has finalized the DFDL v1.0 Specification as an official "Recommendation".
The official PDF is here: http://www.ogf.org/documents/GFD.240.pdf
Added by Michael Beckerle 2021-03-04
The Apache Daffodil project, an Open Source implementation of DFDL v1.0, has graduated from the Apache Incubator, and is now an Apache Top Level Project.
You can read more about this here:
-
ASF "Foundation" blog https://s.apache.org/18vo
Added by Michael Beckerle 2020-11-23
The Apache Daffodil (incubating) community is pleased to announce the release of version 3.0.0.
Notable changes in this release include SAX support, streaming, reduced memory usage, improved diagnostics, and bug fixes.
Detailed release notes and downloads are available at:
https://daffodil.apache.org/releases/3.0.0/
Daffodil is an open source implementation of the Data Format Description Language (DFDL) specification that uses DFDL schemas to parse fixed format data into an infoset, which is most commonly represented as either XML or JSON. This allows the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of the reverse by serializing or "unparsing" an XML or JSON infoset back to the original data format.
For more information about Daffodil visit:
Regards, The Apache Daffodil Team
=====
Disclaimer
Apache Daffodil is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
Added by Michael Beckerle 2020-10-31
Official announcement of this milestone, and links for the public commenting are in the thread below from Greg Newby, OGF Standards Editor
---------- Forwarded message --------- From: Greg Newby gbnewby@petascale.org Date: Sat, Oct 31, 2020 at 12:56 AM Subject: [wg-all] New OGF document in public comment: DFDL 1.0 To: All WG wg-all@ogf.org
Dear OGF Community: A new OGF document is now available for public comment. Please consider reading it and providing your input. Brief affirmations are very useful to demonstrate community input and support. More detailed analysis is, of course, welcome.
DFDL 1.0
The document (Word and PDF) and issue tracker are online at: https://github.com/OpenGridForum/Editor/tree/main/DFDL%201.0%20(GFD%20240)
This is a Recommendation Track document, with a 60-day public comment period.When published, it will obsolete GFD #s 174 and 207.
Abstract: This document provides a definition of astandard Data Format Description Language (DFDL). This language allows description of text,dense binary, and legacy data formats in a vendor-neutral declarative manner.DFDL is an extension to the XML Schema Description Language (XSD). This is the first public comment period for which we are utilizing a new github-based workflow. Public comments may be submitted as "issues" via the github link above. All contributions are subject to the OGF community guidelines as described in the OGF Intellectual Property Policy, online at https://www.ogf.org/ogf/doku.php/about/copyright.
- Greg Newby, OGF Standards Editor
Added by Michael Beckerle about 2020-10-21
ApacheCon 2020 was a virtual conference this year. Mike Beckerle gave a talk about DFDL/Daffodil entitled "Kill the Data Format Problem" a recording of which is available on youtube via this link: https://s.apache.org/apachecon-2020-kill-the-data-format-problem.
The cybersecurity use case for DFDL is described starting at time 18:12, which explains how DFDL is used to address the "Bad-Data Denial-of-Service" threat.
Added by Michael Beckerle 2020-07-15
The Apache Daffodil (incubating) community is pleased to announce the release of version 2.7.0.
Notable changes in this release are primarily the bug fixes, and a small set of useful new features from the DFDL specification.
Detailed release notes and downloads are available at:
https://daffodil.apache.org/releases/2.7.0/
Daffodil is an open source implementation of the Data Format Description Language (DFDL) specification that uses DFDL schemas to parse fixed format data into an infoset, which is most commonly represented as either XML or JSON. This allows the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of the reverse by serializing or "unparsing" an XML or JSON infoset back to the original data format.
For more information about Daffodil visit:
Regards, The Apache Daffodil Team
Added by Michael Beckerle 2020-04-24
The Apache Daffodil (incubating) community is pleased to announce the release of version 2.6.0.
Notable changes in this release include performance and API updates, and the addition of Sonarqube analysis to our workflow.
Detailed release notes and downloads are available at:
https://daffodil.apache.org/releases/2.6.0/
Daffodil is an open source implementation of the Data Format Description Language (DFDL) specification that uses DFDL schemas to parse fixed format data into an infoset, which is most commonly represented as either XML or JSON. This allows the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of the reverse by serializing or "unparsing" an XML or JSON infoset back to the original data format.
For more information about Daffodil visit:
Regards, The Apache Daffodil Team
=====
Disclaimer
Apache Daffodil is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
Added by Michael Beckerle 2020-01-13
The Apache Daffodil (incubating) community is pleased to announce the release of version 2.5.0.
Notable changes in this release include user defined functions, BLOBs, unordered sequences, 2GB+ files, and preparation for decreasing schema compile time.
This release also improves interoperability with IBM DFDL and schemas created for it, including all the schemas published at https://github.com/DFDLSchemas.
Detailed release notes and downloads are available at:
https://daffodil.apache.org/releases/2.5.0/
Daffodil is an open source implementation of the Data Format Description Language (DFDL) specification that uses DFDL schemas to parse fixed format data into an infoset, which is most commonly represented as either XML or JSON. This allows the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of the reverse by serializing or "unparsing" an XML or JSON infoset back to the original data format.
For more information about Daffodil visit:
Regards, The Apache Daffodil Team
Added by Michael Beckerle 2019-07-12
The Apache Daffodil (incubating) community is pleased to announce the release of version 2.4.0.
Notable changes in this release include improved DFDL schema compatibility, new extensions, explicit length choices, better DFDL schema validation, and many bug fixes.
This release improves interoperability with other DFDL implementations by adding the needed DFDL features for support of commercial data formats like EDIFACT, IBM4690TLOG, HL7, ISO8583, and data sets created using COBOL.
Detailed release notes and downloads are available at:
https://daffodil.apache.org/releases/2.4.0/
Daffodil is an open source implementation of the Data Format Description Language (DFDL) specification that uses DFDL schemas to parse fixed format data into an infoset, which is most commonly represented as either XML or JSON. This allows the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of the reverse by serializing or "unparsing" an XML or JSON infoset back to the original data format.
For more information about Daffodil visit:
Added by Michael Beckerle 2019-07-12
DFDL is getting attention in the industrial control / IoT space.
Steve Lawrence of Tresys Technology, will present at this year's ApacheCon NA 2019 in Las Vegas, NV, USA on September 10.
Industrial IoT is made up of many different programmable logic controllers (PLCs) that communicate with often incompatible and proprietary data formats. Apache PLC4X standardizes access to these PLCs by creating a set of libraries across a variety of languages to read and write PLC data with a unified API. Unfortunately, developing and maintaining support for these different data formats, especially across multiple languages, can be burdensome. In this talk we discuss how we can describe PLC data using the open standard Data Format Description Language (DFDL), how Apache Daffodil uses these descriptions to transform data to a standard and easily consumable representation, and an approach to extend PLC4X to use Daffodil and DFDL descriptions to reduce the effort needed to support a variety of PLCs and languages.
https://www.apachecon.com/acna19/s/#/scheduledEvent/1081
Added by Michael Beckerle 2019-06-28
A body of new DFDL tutorial materials have been published at
http://www.xfront.com/DFDL/index.html
This was developed and tested with Apache Daffodil (incubating) by Roger Costello of MITRE Corp. who maintains the XFront site.
The site is XML-oriented and so most of the tutorial material is focused around using DFDL to map non-XML data to/from an XML form. Which is to say the site and this material is for XML enthusiasts. (The Daffodil DFDL implementation also fully supports JSON.)
The tutorial materials cover textual data formats and binary data formats, and the DFDL properties used to parse/unparse them. Lab exercises to help learn DFDL are included as well.
Added by Steve Hanson 2019-05-29
IBM® z/Transaction Processing Facility (z/TPF) is a high-volume, high-throughput transaction processor, that can handle large, continuous loads of transactions across geographically dispersed networks. z/TPF is also an operating system and a unique database, all designed to work together as one system. The z/TPF database and other data formats are highly optimized for transactional workloads, but this can present challenges for other platforms to consume z/TPF data. Data Format Description Language (DFDL) has proven itself to be a robust solution in describing data formats such that the data can be easily consumed by other platforms or from other platforms. DFDL has also proven itself to be very useful in solving data transformation problems. As such, z/TPF has made extensive use of IBM's implementation of DFDL to solve several challenges in an open and extensible manner. Further, z/TPF has created a high performance DFDL parser that is specialized for data formats used by z/TPF.
DFDL enables z/TPF customers to rapidly REST enable their existing services without making application changes. Prior to DFDL support, for each service, customers had to write code to manually parse XML and JSON documents and transform the data into the format the existing application requires. It is time consuming to write and test that code for each service and is often error prone and brittle. With DFDL usage in C/C++ code, the customer uses IBM DFDL tooling to create a DFDL schema for the service. Then the transformation from XML/JSON format to the native application data format is done automatically by the z/TPF system. In the same fashion, DFDL is used to create XML/JSON responses from native application data formats. This has enabled z/TPF customers to dramatically reduce the time and effort required to create and deploy REST services. Further, z/TPF's DFDL parser has been extended to include support to allow transformations to occur to and from native application memory structures composed blocks of memory linked by pointers.
z/TPF customers often send transactional data in real-time to other platforms for multiple business purposes. Transactional data can include data from the transaction itself, database records that were updated by the transaction, as well as application enriched context data to explain why the database updates were made. The data can be put into a stream to feed an event driven architecture system, processed by an analytics platform, or placed in a read-only database (SQL or NoSQL) to be used by other parts of the customer business. Prior to DFDL support, this data would have to be transformed from various formats on z/TPF into a form consumable by the other platform in a process known as Extract Transform and Load (ETL). As part of this ETL processing, fields may need to be hidden, transformed, etc. based upon the use case. A further challenge of these types of scenarios is keeping the various systems in sync regarding the format of the data as the database on z/TPF grows and changes over time. This ultimately results in a lot of custom code being written and maintained. DFDL substantially simplifies this by automatically transforming the z/TPF data, omitting fields, dealing with various data format versions, and more. DFDL can be used to format the transferred data as XML or JSON depending upon the use case. The investment to leverage DFDL has substantially decreased IBM and customer investment in solutions to make z/TPF data available for processing by other systems.
z/TPF MongoDB interface support allows z/TPF customers to access data in z/TPF databases using a standard MongoDB client. Instead of writing a custom data handler, the DFDL parser on z/TPF was modified to operate upon BSON data to transform the data to and from the z/TPF database format. This MongoDB interface provides a standard NoSQL interface to read and write proprietary formatted data on z/TPF where the application developer does not have to know anything about z/TPF and requires no user code on z/TPF. This reduces the development costs and time to market for cloud-based applications that need to access and update data residing on z/TPF.
Often, customers need their z/TPF applications to conditionally send a signal and data to an event consumer system in their enterprise. For example, if an airplane’s inventory is exhausted for given route, the z/TPF system may need to send various information to another system in the enterprise. Signal events provides a transparent configurable infrastructure that the application code can call to send the message off platform. DFDL is used to transform the outgoing message into XML or JSON. In this way, z/TPF customers can quickly code, test, and deploy signaling mechanisms that can be easily consumed by their existing event consumers.
z/TPF customers want to progressively modernize their applications by adding new business logic written in Java that needs to call or be invoked by existing application code written in C/C++. Instead of writing custom connectors and transformations for every possible call, DFDL is used to map C structures to Java objects, and vice-versa. This provides a seamless interface that allows programmers to mix new (Java) and old (C/C++) services within an application to easily extend existing applications and build new Java applications that leverage existing assets.
IBM has implemented various z/TPF performance monitoring tools to aid customers in understanding how their z/TPF systems are running. Some of this tooling now leverages DFDL. For performance reasons, the z/TPF system streams the data in a raw binary format to some media for post processing. This raw data is then consumed by a remote system leveraging IBM DFDL and JAXB to directly convert the binary data into Java objects for further analysis. This is entirely accomplished without writing any parsing code, which substantially reduces the time required to code, test and deliver complex solutions that rely on binary z/TPF data.
z/TPF has found other ways of using DFDL on the system. For example, a tool was written very quickly to dump in memory tables to a CSV file simply by having a DFDL file that described the in-memory table. This provides an easily produced set of diagnostic data in a standard format. DFDL can also be used to format memory for other tooling uses. And as already mentioned, DFDL has a breadth of uses in being able to transform or serialize binary data to and from JSON, XML, etc.
DFDL has become a cornerstone of z/TPF to consume, exchange, and transform data. The DFDL and z/TPF infrastructure provides tremendous development and maintenance efficiencies along with quality improvements for IBM and our customers.
Bradd Kadlecik (IBM z/TPF)
Added by Michael Beckerle 2018-09-27
On Wednesday Sept 26, Mike Beckerle presented DFDL and the Apache Daffodil (Incubating) project which implements DFDL, at the annual ApacheCon North-America conference which was held in at the Marriott Chateau Champlain Hotel, in Montreal, Quebec, Canada.
The slides and audio of the session are available. Slides (as a zipped pdf file) are here: https://s.apache.org/apacheconNA2018-dfdl The audio is here: https://s.apache.org/apacheconNA2018-dfdl-audio
Added by Michael Beckerle 2018-09-05
The Apache Daffodil (incubating) community is pleased to announce the release of version 2.2.0.
Release notes and downloads are available at:
https://daffodil.apache.org/releases/2.2.0/
Daffodil is an open source implementation of the DFDL (Data Format Description Language) specification that uses DFDL schemas to parse fixed format data into an infoset, which is most commonly represented as either XML or JSON. This allows the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of the reverse by serializing or "unparsing" an XML or JSON infoset back to the original data format.
For more information about Daffodil visit:
Regards, The Apache Daffodil Team
Added by Michael Beckerle 2018-05-16
The Apache Daffodil (incubating) community is pleased to announce the release of version 2.1.0.
Release notes and downloads are available at:
https://daffodil.apache.org/releases/2.1.0/
Daffodil is an open source implementation of the DFDL (Data Format Description Language) specification that uses DFDL schemas to parse fixed format data into an infoset, which is most commonly represented as either XML or JSON. This allows the use of well-established XML or JSON technologies and libraries to consume, inspect, and manipulate fixed format data in existing solutions. Daffodil is also capable of the reverse by serializing or "unparsing" an XML or JSON infoset back to the original data format.
For more information about Daffodil visit:
Regards, The Apache Daffodil Team
Added by Steve Hanson 2018-03-08
Following on from their Java implementation DFDL4S, The European Space Agency have released a C++ version. See https://eop-cfi.esa.int/index.php/79-applications/213-new-release-announcement-dfdl4s-library-v1-0
Added by Michael Beckerle 2017-09-06
We are pleased to announce the release of Daffodil 2.0.0!
Downloads are available as zip, tar, and rpm at:
https://opensource.ncsa.illinois.edu/projects/artifacts.php?key=DFDL&version=2.0.0
Daffodil is the open source implementation of the Open Grid Forum's Data Format Description Language (DFDL) specification.
DFDL is a language capable of describing many data formats, including textual and binary, commercial record-oriented, scientific and numeric, modern and legacy, and many industry standards. It leverages XML technology and concepts, using a subset of W3C XML schema type system and annotations to describe such data formats. Various DFDL schemas are publicly available on github at:
https://github.com/DFDLSchemas
Daffodil uses such descriptions to parse data into an XML or JSON infoset for ingestion and validation. Daffodil can also perform the reverse, by serializing or "unparsing" an infoset back to the original data format.
Daffodil 2.0.0 is a monumental milestone that denotes stability and usability, and is ready for general availability. Major features that make up this release include:
- Support for unparse with one-to-one feature parity with parse
- New API with native support for JDOM, w3c DOM, Scala XML, XML text, and JSON
- Support for many non-byte size encodings
- Support for xs:choice direct dispatch
- Support for xs:boolean types
- Support for xs:ignoreCase="yes"
- Support for xs:union restrictions
- Many improvements to the DFDL expression language
- Numerous bug fixes and performance improvements
More details of the changes are available:
https://opensource.ncsa.illinois.edu/confluence/display/DFDL/Changelog
https://opensource.ncsa.illinois.edu/jira/secure/ReleaseNote.jspa?projectId=10010&version=13790
Daffodil 2.0.0 has also been integrated into Apache NiFi and XML Calabash, with the source available in the following git repositories:
Apache NiFi: https://opensource.ncsa.illinois.edu/bitbucket/projects/DFDL/repos/daffodil-nifi/browse
XML Calabash: https://opensource.ncsa.illinois.edu/bitbucket/projects/DFDL/repos/daffodil-calabash-extension/browse
Although Daffodil has reached this milestone, active development continues, with some features not yet available. For a detailed list of unsupported features, visit the unsupported features and errata pages:
https://opensource.ncsa.illinois.edu/confluence/display/DFDL/Unsupported+Features
https://opensource.ncsa.illinois.edu/confluence/display/DFDL/Unsupported+Errata
For more information on Daffodil, visit the Daffodil wiki:
https://opensource.ncsa.illinois.edu/confluence/display/DFDL/
For questions and support, the development team is available on the Daffodil mailing list:
http://oss.tresys.com/mailman/listinfo/daffodil-users
as well as XMPP HipChat:
http://hipchat.ncsa.illinois.edu/gvZdmJHmq
If you are interested in becoming a Daffodil contributor, visit the daffodil wiki or the developers mailing list:
https://opensource.ncsa.illinois.edu/confluence/display/DFDL/
http://oss.tresys.com/mailman/listinfo/daffodil-dev
For a complete list of all open tickets, visit the Daffodil bug tracker:
https://opensource.ncsa.illinois.edu/jira/browse/DFDL/
Thanks to everyone that has helped to make this milestone possible!
Added by Michael Beckerle 2017-07-25
The Daffodil open source DFDL project team is pleased to announce the third release candidate of Daffodil 2.0.0!
Notable changes since the second release candidate include:
- New API with support for JDOM, w3c DOM, Scala XML, XML text, and json
- Support for xs:choice direct dispatch
- Support for xs:boolean types
- Support for 3-, 4-, 5-, and 6-bit encoders
- Support for xs:hexBinary with bit lengths
- Support for xs:ignoreCase="yes"
- Support for xs:union restrictions
- Improvements to the DFDL expression language, including fixes for div/idiv, type conversions and literal constants, bugs in daf:trace(), dfdl:valueLength() and dfdl:contentLength(), and support for fn:error() and math:pow() functions
- Numerous bug fixes and performance improvements
The release candidate has been tagged as 2.0.0-rc3 in the git repo:
https://opensource.ncsa.illinois.edu/bitbucket/projects/DFDL/repos/daffodil/browse
The release is also available for downloaded from the Daffodil Artifacts page:
https://opensource.ncsa.illinois.edu/projects/artifacts.php?key=DFDL&version=2.0.0-RC3
Please test the release candidate and let us know if you find any issues. To report bugs or ask any questions, the development team is available on the Daffodil users mailing list:
http://oss.tresys.com/mailman/listinfo/daffodil-users
as well as XMPP HipChat:
http://hipchat.ncsa.illinois.edu/gvZdmJHmq
or via the JIRA Daffodil bug tracker:
https://opensource.ncsa.illinois.edu/jira/projects/DFDL/summary
Thanks! The Daffodil Team
Added by Michael Beckerle 2017-01-12
Tresys Technology is pleased to announce the release of a DFDL schema to describe MIL-STD-2500C: National Imagery Transmission Format, more commonly known as NITF. The schema and test files are available for download on the DFDLSchemas github page https://github.com/DFDLSchemas/NITF
This schema was developed and tested using the Daffodil open-source DFDL implementation.
The version of Daffodil that has the functionality to parse and unparse NITF files using this schema is still pre-release, but the forthcoming Daffodil 2.0.0 release will incorporate the capability. (Daffodil 2.0.0 development snapshots newer than 2017-01-12 can also be used.)
Added by Michael Beckerle 2017-01-11
Tresys Technology is pleased to announce that the Daffodil open source DFDL implementation is now able to run the NACHA Electronic Payment messages DFDL schema.
This demonstrates an increase in interoperability for Daffodil with DFDL schemas created and tested on the IBM DFDL implementation.
The version of Daffodil that has this functionality is still pre-release, but the forthcoming Daffodil 2.0.0 release will incorporate this capability. (Daffodil 2.0.0 development snapshots newer than 2017-01-11 can also be used.)
The NACHA Electronic Payment messages DFDL schema was originally created and tested by IBM in 2014.
Added by Michael Beckerle 2016-09-28
Tresys and the Daffodil project are pleased to announce the first release candidate of Daffodil 2.0.0, the open source implementation of DFDL v1.0
Notable changes since Daffodil 1.1.0 include unparse achieving full feature parity with parse, support for additional unparse only properties such as outputValueCalc, fillByte, truncateSpecifiedLengthString, etc., numerous bug fixes in parse, and improved parse performance.
The release candidate has been tagged as 2.0.0-rc1 in the git repo:
https://opensource.ncsa.illinois.edu/bitbucket/projects/DFDL/repos/daffodil/browse
The release is also available for downloaded from the Daffodil Artifacts page:
https://opensource.ncsa.illinois.edu/projects/artifacts.php?key=DFDL&version=2.0.0-rc1
Daffodil will now enter the test phase, which we expect to last approximately 3 weeks, followed by the official release of Daffodil 2.0.0.
Please give the release candidate a test and let us know if you find any issues. To report bugs or ask any questions, the development team is available on the Daffodil users mailing list:
http://oss.tresys.com/mailman/listinfo/daffodil-users
as well as XMPP HipChat:
http://hipchat.ncsa.illinois.edu/gvZdmJHmq
or via the JIRA Daffodil bug tracker:
https://opensource.ncsa.illinois.edu/jira/projects/DFDL/summary
On behalf of the Daffodil team,
- Steve Lawrence
Added by Michael Beckerle 2016-06-07
Everyone agrees about the term "parsing", but the inverse of that, well there has been some controversy. Some people use the term "serialization", others prefer "marshalling". In DFDL we have long used the term "unparsing", but somehow that made people uncomfortable in that it felt like we were making up words that didn't really exist.
Well, it was recently discovered that Wiktionary and Wikipedia provide definitions.
These aren't the Oxford English dictionary, but they represent sufficient usage and agreement that we're going to stop apologizing for the term unparsing.
Added by Michael Beckerle 2015-03-03
Tresys Technology, LLC is pleased to announce the release of Daffodil 1.0! Binaries are available at:
https://opensource.ncsa.illinois.edu/projects/artifacts.php?key=DFDL
Daffodil is the open source implementation of the Open Grid Forum's Data Format Description Language (DFDL) specification.
DFDL is a language capable of describing many data formats, including textual and binary, commercial record-oriented, scientific and numeric, modern and legacy, and many industry standards. It leverages XML technology and concepts, using a subset of W3C XML schema type system and annotations to describe such data formats. Daffodil uses this description to parse data into an XML infoset for ingestion and validation.
Daffodil 1.0 is a monumental milestone that denotes stability and usability, and is ready for general availability. Major features that make up this release include:
- Command line interface available for Linux, Windows, and Mac
- API available in both Java and Scala
- Interactive debugger to incrementally step through a parse
- Test harness using Test Data Markup Language (TDML)
- Ability to save and reload Daffodil parsers
- Support for parsing both textual and binary data
- Unbounded lookahead with backtracking guided by asserts and discriminators
- Numeric representations in text and two's complement binary
- Reusable definitions of formats and escape schemes
- DFDL expressions and variables for dynamic formats
- Delimited text with initiators, terminators, and separators
- Text justification (left, right, or center) with padding
- Binary data alignment, byte order, and bit order
- Fixed and variable data lengths
- Simple types like string, integer, floating point, hexBinary, date, and time
- Ordered sequences and choices
- Arrays with fixed or variable repetitions
- XSD facets and restrictions for validation
Although Daffodil has reached this milestone, Daffodil continues active development, with some features not yet available. Changes planned for the future include:
- Unparsing support
- Performance improvements
- Unordered sequences
- New variable instances
- Boolean and binary calendar types
- Fixed and default XSD attributes
- IBM Mainframe numeric representations, including packed and zoned binary
- Bi-directional text
- Floating elements
For a more detailed list of unsupported features, visit the unsupported features and errata pages:
https://opensource.ncsa.illinois.edu/confluence/display/DFDL/Unsupported+Features
https://opensource.ncsa.illinois.edu/confluence/display/DFDL/Unsupported+Errata
For a complete list of all open tickets, visit the Daffodil bug tracker:
https://opensource.ncsa.illinois.edu/jira/browse/DFDL/
For more information on Daffodil, visit the Daffodil wiki:
https://opensource.ncsa.illinois.edu/confluence/display/DFDL/
For questions and support, the development team is available on the Daffodil mailing list:
http://oss.tresys.com/mailman/listinfo/daffodil-users
as well as XMPP HipChat:
http://hipchat.ncsa.illinois.edu/gvZdmJHmq
If you are interested in becoming a Daffodil contributor, visit the daffodil wiki or the developers mailing list:
https://opensource.ncsa.illinois.edu/confluence/display/DFDL/
http://oss.tresys.com/mailman/listinfo/daffodil-dev
Thanks to everyone that has helped to make this milestone possible!
Added by Michael Beckerle 2015-02-24
Some public comments were received on this document, and it has been updated accordingly, and is now published as [GFD 216].
This document summarizes the changes to DFDL motivated by the MIL-STD-2045 format which required adding the dfdl:bitOrder property to DFDL to handle "least significant bit first" data. The material in this document has been incorporated into the DFDL v1.0 specification.
Added by Michael Beckerle 2014-12-02
This schema has been updated based on thorough review of the original version. The result is greatly superior. It is far simpler now, and is a much better illustration of how to use the dfdl:bitOrder property, which was added to DFDL in order to support this sort of dense binary data formats.
Conformance with the DFDL specification has also been improved - Some missing properties that are required by the DFDL standard are now present. Also, prior versions were depending upon some artifacts of the Daffodil DFDL implementation that are not conforming (and those bugs have been reported.)
This DFDL schema can be used with the Daffodil DFDL implementation -- where the dfdl:bitOrder feature of DFDL was prototyped and implemented.
Added by Steve Hanson 2014-11-05
A new repository has been added to the DFDLSchemas GitHub website which contains DFDL schemas for the vCard standard.
The initial schemas in the repository model vCard 2.1.
The schemas are compatible with IBM DFDL 1.1.1 release, as shipped in IBM Integration Bus 9.0.0.2 onwards.
Full details in the repository readme at https://github.com/DFDLSchemas/vCard.
Added by Steve Hanson 2014-11-05
The updated DFDL 1.0 specification is http://www.ogf.org/documents/GFD.207.pdf. This obsoletes the original specification GFD.174. Also published are two DFDL experience documents http://www.ogf.org/documents/GFD.214.pdf and http://www.ogf.org/documents/GFD.215.pdf.
Added by Steve Hanson 2014-09-18
Modeling a MIL-STD-2045 header in DFDL v1.0 is not possible without the addition of new capabilities for specifying bit order and non-standard encodings. There are many related military-standard binary data formats which are similar, and so cannot be modeled in DFDL.This document describes the new properties and property values that are required to successfully model this format.
Document and comment forum here: shttp://redmine.ogf.org/projects/editor-pubcom/boards/21
Added by Michael Beckerle 2014-07-31
The DFDL PCAP (packet capture) schema has been updated:
- Now parses parts of the link layer (ethernet) the network layer (IPv4, IPv6, ICMPv4) and the transport layer (TCP, UDP)
- Remove lldp.minimal capture since the new schema does not define lldp
- Add new IPv6 and ICMPv4 test captures
See https://github.com/DFDLSchemas/PCAP
Added by Michael Beckerle 2014-07-31
A preliminary version of a DFDL Schema for MIL-STD-2045 is now available on the github DFDL Schemas site.
This schema makes use of the dfdl:bitOrder='leastSignificantBitFirst' property and the x-dfdl-us-ascii-7-bit-packed character set encoding that are being added to DFDL v1.0 in response to a public comment review.
These features have been prototyped in the Daffodil open-source DFDL implementation; hence, this DFDL schema will (as of this writing) only run with that DFDL implementation. This schema is, in effect, the proof point that the new DFDL features will in fact address the needs that were raised by the public comment on bit ordering.
This data format has much in common with other binary MIL-STD data formats, so techniques illustrated in it are expected to be used in other DFDL schemas. However, as of this news item some improved ways of modeling the data have already been discovered, so this DFDL schema for MIL-STD-2045 is expected to evolve substantially.
The schema is available at https://github.com/DFDLSchemas/mil-std-2045
Added by Steve Hanson 2014-07-31
The repository in DFDLSchemas GitHub which contains DFDL schemas for the Toshiba Commerce (formerly IBM) 4690 Transaction Log (Tlog) point-of-sale format has been updated.
It now includes schemas for SuperMarket Application (SA) in addition to the existing ACE schemas.
The schemas are compatible with IBM DFDL 1.0 and 1.1 releases, as shipped in IBM WebSphere Message Broker 8.0.0.2 onwards and IBM Integration Bus 9.0.0.1 onwards, respectively.
Example data streams and XML infosets are included.
Full details in the repository readme at https://github.com/DFDLSchemas/IBM4690-TLOG.
Added by Steve Hanson 2014-05-14
The repository in DFDLSchemas GitHub which contains DFDL schemas for the ISO 8583 standard has been updated.
It now includes schemas for 1993 and 1987 versions of the standard, supporting both packed bitmaps and unpacked bitmaps.
The schemas are compatible with IBM DFDL 1.0 and 1.1 releases, as shipped in IBM WebSphere Message Broker 8.0.0.2 onwards and IBM Integration Bus 9.0.0.1 onwards, respectively.
Example data streams and XML infosets are included.
Full details in the repository readme at https://github.com/DFDLSchemas/ISO8583.
Added by Steve Hanson 2014-03-05
A new repository has been added to the DFDLSchemas GitHub website which contains DFDL schemas for the UN/EDIFACT standard.
The initial schemas in the repository model a selection of supply chain messages from the D03B release, namely DESADV, INVOIC, ORDCHG, ORDERS, ORDRSP, RECADV and REMADV. Other messages and releases are available from IBM.
The schemas are compatible with IBM DFDL 1.0 and 1.1 releases, as shipped in IBM WebSphere Message Broker 8.0.0.2 onwards and IBM Integration Bus 9.0.0.1 onwards, respectively.
The schemas include the following features:
Full UN/EDIFACT interchange;
Dynamically defined delimiters via UNA segment;
Syntax version 4 and syntax version 3 (minor edit required);
Segments with or without new lines at the end;
Interactive messages via UIH/UIT segments;
AUTACK, CONTRL and KEYMAN service messages;
Separate schemas for each industry/release combination, in their own namespace.
Full details in the repository readme at https://github.com/DFDLSchemas/EDIFACT.
Added by Steve Hanson 2013-01-30
Our public DFDL schemas web community is now live. The first DFDL schemas are up there, others will follow. Anybody can download and use the schemas subject to accompanying license. Anybody can apply to the DFDL WG to create a repository for a new format, or to contribute to an existing repository.
Web page: http://dfdlschemas.github.com/ Organization: https://github.com/DFDLSchemas