draft-netana-nmop-yang-message-broker-integration-00.xml

<?xml version="1.0" encoding="US-ASCII"?>
<!DOCTYPE rfc SYSTEM "rfc2629.dtd">
<?rfc toc="yes"?>
<?rfc tocompact="yes"?>
<?rfc tocdepth="2"?>
<?rfc tocindent="yes"?>
<?rfc symrefs="yes"?>
<?rfc sortrefs="yes"?>
<?rfc comments="yes"?>
<?rfc inline="yes"?>
<?rfc compact="yes"?>
<?rfc subcompact="no"?>
<rfc category="info"
     docName="draft-netana-nmop-yang-message-broker-integration-00"
     ipr="trust200902">
  <front>
    <title abbrev="YANG-Push to Message Broker Integration">An Architecture
    for YANG-Push to Message Broker Integration</title>

    <author fullname="Thomas Graf" initials="T" surname="Graf">
      <organization>Swisscom</organization>

      <address>
        <postal>
          <street>Binzring 17</street>

          <city>Zurich</city>

          <code>8045</code>

          <country>Switzerland</country>
        </postal>

        <email>thomas.graf@swisscom.com</email>
      </address>
    </author>

    <author fullname="Ahmed Elhassany" initials="A" surname="Elhassany">
      <organization>Swisscom</organization>

      <address>
        <postal>
          <street>Binzring 17</street>

          <city>Zuerich 8045</city>

          <region/>

          <code/>

          <country>Switzerland</country>
        </postal>

        <phone/>

        <email>ahmed.elhassany@swisscom.com</email>

        <uri/>
      </address>
    </author>

    <date day="22" month="April" year="2024"/>

    <area>Operations and Management</area>

    <workgroup>NMOP</workgroup>

    <abstract>
      <t>This document describes the motivation and architecture of a native
      YANG-Push notifications and YANG Schema integration into a Message
      Broker and YANG Schema Registry.</t>
    </abstract>

    <note removeInRFC="true">
      <name>Discussion Venues</name>

      <t>Discussion of this document takes place on the Operations and
      Management Area Working Group Working Group mailing list
      (nmop@ietf.org), which is archived at <eref
      target="https://mailarchive.ietf.org/arch/browse/nmop/"/>.</t>

      <t>Source for this draft and an issue tracker can be found at <eref
      target="https://github.com/network-analytics/draft-daisy-kafka-yang-integration/"/>.</t>
    </note>
  </front>

  <middle>
    <section anchor="Introduction" title="Introduction">
      <t>Nowadays network operators are using <xref
      target="RFC7950">YANG</xref> to model their configurations and obtain
      YANG modelled data from their networks. It is well understood that plain
      text are initially intended for humans and need effort to make it
      machine readable due to the lack of semantics. YANG modeled data is
      addressing most of these needs.</t>

      <t>Increasingly more network operators organizing their data in a <xref
      target="Deh22">Data Mesh</xref> where a Message Broker such as <xref
      target="Kaf11">Apache Kafka</xref> or <xref
      target="Rab07">RabbitMQ</xref> facilitates the exchange of messages
      among data processing components like a stream processor to filter,
      enrich, correlate or aggregate, or a time series database to store
      data.</t>

      <t>Even though YANG is intend to ease the handling of data, this promise
      has not yet been fulfilled for <xref target="RFC9232">Network
      Telemetry</xref>. From subscribing on a YANG datastore, publishing a
      YANG modeled notifications message from the network and viewing the data
      in a time series database, manual labor is needed to perform a data
      transformation to make a Message Broker and its data processing
      components with YANG notifications interoparable.</t>

      <section anchor="Document_Scope" title="Document Scope">
        <t>This document focuses on <xref target="RFC8641">YANG-Push</xref> as
        the messaging protocol between the network node and the <xref
        target="RFC9232">Network Telemetry</xref> data collection. It
        describes the main components and the aimed architecture for deploying
        such solution in a production network. Then, it illustrates the
        integration of the <xref target="RFC7950"> YANG 1.1</xref> as a schema
        modeling language into the Apache Kafka Message Broker and <xref
        target="Con18"> Confluent Schema Registry</xref>.</t>
      </section>

      <section anchor="Terminology" title="Terminology">
        <t>The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
        "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
        "OPTIONAL" in this document are to be interpreted as described in BCP
        14 <xref target="RFC2119"/> <xref target="RFC8174"/> when, and only
        when, they appear in all capitals, as shown here.</t>

        <t>This document defines the following terms:</t>

        <t>Message Broker: is an intermediary software component that
        translates messages from the formal messaging protocol of the sender
        to the formal messaging protocol of the receiver routed in topics.
        Message brokers are elements in Data Mesh where software applications
        communicate by exchanging formally-defined messages.</t>

        <t>Stream Catalog: provides a single point of access that allows users
        to centrally search semantics for information across a Message
        Broker.</t>

        <t>Additionally it makes use of the terms defined in <xref
        target="RFC8639"/>, <xref target="Kaf11">Apache Kafka</xref> and <xref
        target="ConDoc18">Confluent Schema Registry Documentation</xref>.</t>

        <t>The following terms are used as defined in <xref
        target="RFC8639"/>.</t>

        <t><list style="symbols">
            <t>Publisher</t>

            <t>Receiver</t>

            <t>Subscription</t>

            <t>Subscription ID</t>

            <t>Event stream filter</t>

            <t>Notification message</t>
          </list></t>

        <t>The following terms are used as defined in <xref
        target="Kaf11">Apache Kafka Message Broker</xref>.</t>

        <t><list style="symbols">
            <t>Producer</t>

            <t>Consumer</t>

            <t>Topic</t>

            <t>Partition</t>
          </list></t>

        <t>The following terms are used as defined in <xref
        target="ConDoc18">Confluent Schema Registry Documentation</xref>.</t>

        <t><list style="symbols">
            <t>Schema</t>

            <t>Schema ID</t>

            <t>Schema Registry</t>

            <t>Subject</t>
          </list></t>
      </section>

      <section anchor="Origins_of_YANG_Push" title="Origins of YANG-Push">
        <t>With <xref target="RFC3535"/> the IAB set the requirements for
        Network Management in 2003. From these requirements <xref
        target="RFC6241">NETCONF</xref>, <xref target="RFC5277">NETCONF
        Notifications</xref> and <xref target="RFC8040">RESTCONF</xref> has
        been defined to configure through &lt;edit-config&gt; and retrieve
        operational data through &lt;get&gt; and NETCONF notifications through
        &lt;notification&gt; from a YANG datastore on a network node.</t>

        <t>With YANG-Push, as defined in <xref target="RFC8639"/>, <xref
        target="RFC8640"/> and <xref target="RFC8641"/>, periodical and
        on-change subscriptions to the YANG datastore can be dynamically or
        statically configured. When notifications are dynamically configured,
        messages are published over the initially established NETCONF session,
        while when it is statically configured messages are published through
        <xref target="I-D.ietf-netconf-https-notif">HTTPS-based</xref> or
        <xref target="I-D.ietf-netconf-udp-notif">UDP-based</xref> transport.
        Section 3.7 of <xref target="RFC8641"/> describes push-update messages
        where the YANG subscribed data is being published, where Section 2.7
        of <xref target="RFC8639"/> describes the subscription state change
        notifications where changes in the subscription are being
        described.</t>
      </section>

      <section anchor="Origins_of_Apache_Kafka"
               title="Origins of Apache Kafka">
        <t><xref target="Kaf11">Apache Kafka</xref> is a Message Broker that
        supports producing and consuming messages from so called topics. Each
        topic has one or more partitions where messages are replicated or load
        balanced to scale out. With the introduction of <xref
        target="Con18">Confluent Schema Registry</xref> a topic can contain
        one or more subjects. A subject refers to a Schema defining the
        structure of the message. The Schema then is used to validate messages
        sent through topics and are identified by a Schema ID. The Schema ID
        is issued when the Schema is registered to the Confluent Schema
        Registry. Once the Schema ID is obtained, it can be prefixed to the
        message with a Confluent Schema Registry compatible serializer.
        Messages can then be validated against Schema at the producer or at
        the consumer from a topic to ensure Schema integrity of the message.
        The type of Schema evolution scheme can be defined per subject,
        wherever non backward compatibility changes are allowed or not.</t>
      </section>
    </section>

    <section anchor="Motivation" title="Motivation">
      <t>There are four main objectives for native YANG-Push notifications and
      YANG Schema integration into a Message Broker.</t>

      <section anchor="Automtaic_Onboarding" title="Automatic Onboarding">
        <t>Automate the Data Mesh onboarding of newly subscribed YANG
        metrics.</t>
      </section>

      <section anchor="Preserve_Schema" title="Preserve Schema">
        <t>The preservation of the YANG schema, that includes the YANG data
        types as defined in <xref target="RFC6991"/> and the nested structure
        of the YANG module, throughout the data processing chain ensures that
        metrics can be processed and visualized as they were originally
        intended. Not only for the user but also for an automated closed loop
        operation action.</t>
      </section>

      <section anchor="Preserve_Semantic_Information"
               title="Preserve Semantic Information">
        <t><xref target="RFC7950"/> defines in Section 7.21.3 and 7.21.4 the
        description and reference statement. This information is intended for
        the user, describing in a human-readable fashion the meaning of a
        definition. In Data Mesh, this information can be imported from the
        YANG Schema Registry into a Stream Catalog where subjects within
        Message Broker are identifiable and searchable. An example of a Stream
        Catalog is <xref target="Atl15">Apache Atlas</xref>. It can also be
        applied for time series data visualization in a similar fashion.</t>
      </section>

      <section anchor="Standardize_Data_Processing_Integration"
               title="Standardize Data Processing Integration">
        <t>Since the YANG Schema is preserved for operational metrics in the
        Message Broker, a standardization for integration between network data
        collection and stream processor or time series database is
        implied.</t>
      </section>
    </section>

    <section anchor="Elements_of_the_Architecture"
             title="Elements of the Architecture">
      <t>The architecture consists of 6 elements. <xref
      target="workflow_diagram"/> gives an overview on the workflow.</t>

      <figure anchor="workflow_diagram" title="End to End Workflow">
        <artwork align="center"><![CDATA[
   +------------------------------------------------------------+
   |                    Time Series Database                    |
   +------------------------------------------------------------+
                                  ^
                                  | (11) Ingest Data
                                  | According to Schema
   +------------------------------------------------------------+
   |                Time Series Database Ingestion              |
   +------------------------------------------------------------+
(9) Get  |  ^                                   ^ (8) Validate serialized Message
 Schema  |  |                                   | Against Schema on Consumer
         |  |                                   |
         |  |                                   |
         |  | (10) Issue                        | (7) Serialize YANG-Push Message
         v  | Schema             (5) Post       | annotated Schema ID
   +--------------------+          Schema  +--------------------+
   |       YANG         | <--------------  |  Data Collection   |
   |  Schema Registry   | -------------->  | YANG-Push Receiver |
   +--------------------+ (6) Issue        +--------------------+
                          Schema ID     (3) Get |  ^ (2) Receive YANG-Push
                                         Schema |  | Subscription Start Message
                                                |  |   ^
                                                |  |   |
                                                |  |   | (4) Publish YANG-Push
                                                v  |   | Message with Subscription ID
   +--------------------+                  +--------------------+
   |      Network       | (1) Subscribe    |   Network Node     |
   |   Orchestration    | ---------------> | YANG-Push Publisher|
   +--------------------+                  +--------------------+
]]></artwork>
      </figure>

      <t>The <xref target="workflow_diagram">workflow diagram</xref> describes
      the steps from establishing the YANG-Push subscription to Time Series
      Database ingestion.</t>

      <section anchor="YANG_Push_Subscription" title="YANG-Push Subscription">
        <t>With step number (1) in the workflow diagram, a YANG-Push
        subscription is according to Section 2.4 and 2.5 of <xref
        target="RFC8639"/> dynamically or statically configured, and with step
        (2) subscription state change notifications are sent according to
        section 2.7 from the YANG-Push publisher to the receiver to inform
        which event stream filter has been applied to which subscription
        ID.</t>

        <t><xref target="I-D.ietf-netconf-yang-notifications-versioning"/>
        adds the capability to subscribe to a specific YANG module revision or
        a YANG module which needs to be backward compatible to in step (1) and
        adds the module name, revision and revision-label information into the
        subscription state change notifications in step (2).</t>

        <t><xref
        target="netconf_edit_config_establish_subscription_example_xml_fig"/>
        provides and example how to create a YANG-Push configured subscription
        with NETCONF in XML <xref target="W3C.REC-xml-20081126"/> with
        UDP-based <xref target="I-D.ietf-netconf-udp-notif"/> transport</t>

        <figure anchor="netconf_edit_config_establish_subscription_example_xml_fig"
                title="NETCONF Example to establish configured subscription">
          <artwork><![CDATA[
<rpc message-id="101"
  xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <edit-config>
    <target>
      <running/>
    </target>
    <config>
      <subscriptions xmlns="urn:ietf:params:xml:ns:yang:ietf-subscribed-notifications">
        <subscription>
          <id>6666</id>
          <datastore xmlns="urn:ietf:params:xml:ns:yang:ietf-yang-push"
            xmlns:ds="urn:ietf:params:xml:ns:yang:ietf-datastores">ds:operational</datastore>
          <datastore-xpath-filter xmlns="urn:ietf:params:xml:ns:yang:ietf-yang-push"
            xmlns:if="urn:ietf:params:xml:ns:yang:ietf-interfaces">/if:interfaces</datastore-xpath-filter>
          <revision xmlns="urn:ietf:params:xml:ns:yang:ietf-yang-push-revision">2018-02-20</revision>
          <transport xmlns:unt="urn:ietf:params:xml:ns:yang:ietf-udp-notif-transport">unt:udp-notif</transport>
          <encoding>encode-json</encoding>
          <receivers>
            <receiver>
              <name>subscription-specific-receiver-def</name>
              <receiver-instance-ref xmlns="urn:ietf:params:xml:ns:yang:ietf-subscribed-notif-receivers">global-udp-notif-receiver-def</receiver-instance-ref>
            </receiver>
          </receivers>
          <periodic xmlns="urn:ietf:params:xml:ns:yang:ietf-yang-push">
            <period>6000</period>
          </periodic>
        </subscription>
        <receiver-instances xmlns="urn:ietf:params:xml:ns:yang:ietf-subscribed-notif-receivers">
          <receiver-instance>
            <name>global-udp-notif-receiver-def</name>
            <udp-notif-receiver xmlns="urn:ietf:params:xml:ns:yang:ietf-udp-notif-transport">
              <address>192.0.5.1</address>
              <port>12345</port>
              <enable-segmentation>false</enable-segmentation>
              <max-segment-size/>
            </udp-notif-receiver>
          </receiver-instance>
        </receiver-instances>
      </subscriptions>
    </config>
  </edit-config>
</rpc>
          ]]></artwork>
        </figure>

        <t><xref target="subscription_started_notif_example_json_fig"/>
        provides an example of a JSON encoded, <xref target="RFC7951"/>,
        subscription-started state change notification message over
        HTTPS-based <xref target="I-D.ietf-netconf-https-notif"/> or UDP-based
        <xref target="I-D.ietf-netconf-udp-notif"/> transport for the same
        subscription.</t>

        <figure anchor="subscription_started_notif_example_json_fig"
                title="JSON YANG-Push Example for a subscription-started notification message">
          <artwork><![CDATA[
{
  "ietf-notification:notification": {
    "eventTime": "2023-03-25T08:30:11.22Z",
    "ietf-notification-sequencing:sysName": "example-router",
    "ietf-notification-sequencing:sequenceNumber": 1,
    "ietf-subscribed-notification:subscription-started": {
      "id": 6666,
      "ietf-yang-push:datastore": "ietf-datastores:operational",
      "ietf-yang-push:datastore-xpath-filter": "/if:interfaces",
      "ietf-yang-push-revision:revision": "2014-05-08",
      "ietf-yang-push-revision:module-name": "ietf-interfaces",
      "ietf-yang-push-revision:revision-label": "",
      "ietf-distributed-notif:message-observation-domain-id": [1,2],
      "transport": "ietf-udp-notif-transport:udp-notif",
      "encoding": "encode-json",
      "ietf-yang-push:periodic": {
        "ietf-yang-push:period": 100
      }
    }
  }
}
          ]]></artwork>
        </figure>
      </section>

      <section anchor="YANG_Push_Publisher" title="YANG-Push Publisher">
        <t>With step number (4) in the workflow diagram, a YANG-Push
        push-update or push-change-update message, depending on wherever
        periodical or on-change subscription has been established, is sent
        from the YANG-Push publisher to the receiver according to Section 3.7
        of <xref target="RFC8639"/>.</t>

        <t><xref target="I-D.ahuang-netconf-notif-yang"/> defines the NETCONF
        notification header specified in <xref target="RFC5277"/> in YANG to
        enable JSON and CBOR encoding.</t>

        <t><xref target="I-D.tgraf-netconf-notif-sequencing"/> adds sysName,
        messagePublisherId and sequenceNumber in the NETCONF notification
        header to each message to identify from which network node and
        publishing process, according to <xref
        target="I-D.ietf-netconf-distributed-notif"/> a network node with
        distributed architecture could have multiple messagePublisherId's, the
        message has been published from. The sequenceNumber enables to
        recognize loss from the YANG-Push publisher in step (1) down to the
        Time Series Database Ingestion in step (11).</t>

        <t><xref target="I-D.tgraf-netconf-yang-push-observation-time"/> adds
        observation-time or state-changed-observation-time in the YANG-Push
        push-update or push-change-update message, depending on wherever
        periodical or on-change subscription has been established.
        observation-time describes when the operational metrics was obtained
        from the YANG datastore. Where state-changed-observation-time
        describes when the network state change was observed.</t>

        <t><xref target="push_update_notif_example_json_fig"/> provides an
        example of a JSON encoded, <xref target="RFC7951"/>, push-update
        notification message over HTTPS-based <xref
        target="I-D.ietf-netconf-https-notif"/> or UDP-based <xref
        target="I-D.ietf-netconf-udp-notif"/> transport for the same
        subscription.</t>

        <figure anchor="push_update_notif_example_json_fig"
                title="JSON YANG-Push Example for a push-update notification message">
          <artwork><![CDATA[
{
  "ietf-notification:notification": {
    "eventTime": "2023-03-25T08:30:11.22Z",
    "ietf-notification-sequencing:sysName": "example-router",
    "ietf-notification-sequencing:sequenceNumber": 1,
    "ietf-yang-push:push-update": {
      "id": 6666,
      "ietf-yang-push-netobs-timestamping:observation-time": "2023-03-25T08:30:11.22Z",
      "datastore-contents": {
        "ietf-interfaces:interfaces": [
          {
            "interface": {
              "name": "eth0",
              "type": "iana-if-type:ethernetCsmacd",
              "oper-status": "up",
              "mtu": 1500
            }
          }
        ]
      }
    }
  }
}     
           ]]></artwork>
        </figure>

        <t><xref target="push_change_update_notif_example_json_fig"/> provides
        an example of a JSON encoded, <xref target="RFC7951"/>,
        push-change-update notification message over HTTPS-based <xref
        target="I-D.ietf-netconf-https-notif"/> or UDP-based <xref
        target="I-D.ietf-netconf-udp-notif"/> transport for the same
        subscription.</t>

        <figure anchor="push_change_update_notif_example_json_fig"
                title="JSON YANG-Push Example for a push-change-update notification message">
          <artwork><![CDATA[
{
  "ietf-notification:notification": {
    "eventTime": "2023-03-25T08:30:11.22Z",
    "ietf-notification-sequencing:sysName": "example-router",
    "ietf-notification-sequencing:sequenceNumber": 1,
    "ietf-yang-push:push-change-update": {
      "id": 2222,
      "ietf-yang-push-netobs-timestamping:state-changed-observation-time": "2023-03-25T08:30:11.22Z",
      "datastore-contents": {
        "yang-patch": {
          "patch-id": "patch_54",
          "comment": "Changing encoding to JSON and increasing the period to 10 minutes",
          "edit": [
            {
              "edit-id": "id_change_1",
              "operation": "merge",
              "target": "/ietf-subscribed-notifications:subscriptions/subscription[id=2222]",
              "value": {
                "ietf-subscribed-notifications:encoding": "ietf-subscribed-notifications:encode-json",
                "ietf-yang-push:periodic": {
                  "period": 60000
                }
              }
            }
          ]
        }
      }
    }
  }
}    
           ]]></artwork>
        </figure>
      </section>

      <section anchor="YANG_Push_Receiver" title="YANG-Push Receiver">
        <t>For all the YANG modules and revisions of each subscription ID in
        the subscription state change notification received in step number (3)
        in the workflow diagram, all the YANG module dependencies need to be
        determined through the <xref target="RFC8525">YANG Library</xref>, and
        then through NETCONF &lt;get-schema&gt; rpc calls according to <xref
        target="RFC6022"/> all YANG modules need to be retrieved as described
        in step (3) in the workflow diagram.</t>

        <t><xref target="I-D.lincla-netconf-yang-library-augmentation"/>
        extends the YANG Library so that not only the submodule but also the
        augmentation list can be obtained.</t>

        <t>Figure 9 in Section 4.1 and YANG module in Section 5 of <xref
        target="RFC8641"/> defines the payload of YANG-push notifications
        where "datastore-contents" or the "value" of a "push-change-update")
        is "anydata". <xref target="RFC7950"/> Section 7.10 states that
        anydata represents an unknown set of nodes that can be modeled with
        YANG, and the model is not known at design time and that the model of
        the unknown set of nodes can be signaled through another protocol.
        This poses and issue in the schema validation of YANG-Push
        notifications and will be further clarified in point number (1) and
        (2) in <xref target="Open_Points"/>.</t>
      </section>

      <section anchor="YANG_Schema_Registry" title="YANG Schema Registry">
        <t>The schema registry SHOULD support YANG as the format for defining
        schema. For each schema registered into the schema registry, a schema
        ID is returned. That schema ID can be used when interacting with the
        Message Broker to indicate the schema to use with the
        message.&rdquo;</t>

        <t>Confluent Schema Registry is pluggable. Currently Supports AVRO,
        JSON Schema and Protobuf. The YANG support is being developed at <xref
        target="Yak24"/> as part of this architecture. Enable to register,
        obtain and compare <xref target="YSR24"/> YANG Schemas. One YANG
        Schema with all its augmentations is being registered per YANG-Push
        subscription ID. for each YANG Schema a locally significant Schema ID
        is being issued as described in step (6) in the workflow diagram.</t>

        <figure anchor="YSR_post_ietf-interfaces"
                title="Register ietf-interfaces.yang into YANG Schema Registry">
          <artwork><![CDATA[
curl -X POST -H "Content Type: application/vnd.schemaregistry.v1+json"
-d @ietf-interfaces@2018-02-20.json 
http://localhost:8081/subjects/ietf-interfaces/
        ]]></artwork>
        </figure>

        <figure anchor="YSR_list_all_subjects"
                title="List all subjects YANG Schema Registry">
          <artwork align="left"><![CDATA[
curl http://localhost:8081/subjects/ ubjects/ | jq
        ]]></artwork>
        </figure>

        <figure anchor="YSR_list_versions_of_a_subject"
                title="List versions of a given subject in YANG Schema Registry">
          <artwork><![CDATA[
curl http://localhost:8081/subjects/ietf-interfaces/versions
        ]]></artwork>
        </figure>

        <figure anchor="YSR_retrieve_schema_of_subject_version"
                title="Retrieve schema of a specific subject and version in YANG Schema Registry">
          <artwork><![CDATA[
curl http://localhost:8081/subjects/ietf-interfaces/versions/1
        ]]></artwork>
        </figure>
      </section>

      <section anchor="YANG_Message_Broker_Producer"
               title="YANG Message Broker Producer">
        <t>The previously issued Schema ID is prefixed to the previously in
        <xref target="YANG_Push_Receiver"/> described metadata augmented YANG
        push push-update message and serialized to a Message Broker topic in
        step (7) of the workflow diagram.</t>
      </section>

      <section anchor="YANG_Message_Broker_Consumer"
               title="YANG Message_Broker Consumer">
        <t>From the Message Broker topic the message is being consumed and the
        prefixed Schema ID is being used in step (9) of the workflow diagram
        to retrieve the YANG Schema to validate the Schema integrity of the
        message.</t>
      </section>

      <section anchor="YANG_Time_Series_Database_Ingestion"
               title="YANG Time Series Database Ingestion">
        <t>The time series database ingestion specifications are being derived
        with the in <xref target="YANG_Message_Broker_Consumer"/> already
        retrieved Schema ID and YANG-Push push-update messages can be now
        ingested and indexed into the database table according to their
        schema.</t>
      </section>
    </section>

    <section anchor="Project_Milestones" title="Project Milestones">
      <t>IETF 115:</t>

      <ul>
        <li>Official Project Kickoff.</li>

        <li><xref target="I-D.ietf-netconf-yang-notifications-versioning"/>
        extends schema reference in subscription state change
        notification.</li>
      </ul>

      <t>IETF 116:</t>

      <ul>
        <li>YANG module with augmentations can be registered in Confluent
        Schema Registry with YANG extension <xref target="Yak24"/>.</li>

        <li><xref target="I-D.tgraf-netconf-notif-sequencing"/> extends
        NETCONF notification header with sysName, messagePublisherId and
        sequenceNumber.</li>

        <li><xref target="I-D.tgraf-netconf-yang-push-observation-time"/>
        extends YANG-Push push-update or push-change-update message with
        observation-time or state-changed-observation-time.</li>

        <li><xref target="I-D.ahuang-netconf-notif-yang"/> defines the NETCONF
        notification header specified in <xref target="RFC5277"/> in
        YANG.</li>
      </ul>

      <t>IETF 118:</t>

      <ul>
        <li>All relevant YANG modules for a subscribed xpath can be determined
        through the <xref target="RFC8525">YANG Library</xref> and retrieved
        throug NETCONF &lt;get-schema&gt; rpc calls according to <xref
        target="RFC6022"/>. Gap in YANG library addressed in <xref
        target="I-D.lincla-netconf-yang-library-augmentation"/>.</li>
      </ul>

      <t>IETF 119:</t>

      <ul>
        <li><xref target="I-D.aelhassany-anydata-validation"/> addresses that
        anydata modeled nodes can be validated with YANG Library RFC 8525.
        6WIND VSR and Huawei VRP YANG-Push publisher and open-source <xref
        target="I-D.lincla-netconf-yang-library-augmentation"/> implementation
        validated at hackathon.</li>
      </ul>

      <t>IETF 120:</t>

      <ul>
        <li>6WIND VSR, Huawei VRP and Cisco IOS XR YANG-Push publisher and
        <xref target="I-D.aelhassany-anydata-validation"/> implementation
        validated at hackathon.</li>
      </ul>
    </section>

    <section anchor="Open_Points" title="Open Points">
      <t>Lists all current open points to be either further researched and
      clarified or tested with running code.</t>

      <t>Note to the RFC-Editor: Please remove this section before
      publishing.</t>

      <dl newline="true">
        <dt>Open Point 1:</dt>

        <dd>Figure 9 in Section 4.1 and YANG module in Section 5 of <xref
        target="RFC8641"/> defines the payload of YANG-push notifications
        where "datastore-contents" or the "value" of a "push-change-update")
        is "anydata". <xref target="RFC7950"/> Section 7.10 states that
        anydata represents an unknown set of nodes that can be modeled with
        YANG, and the model is not known at design time and that the model of
        the unknown set of nodes can be signaled through another protocol. How
        to exchange the anydata modeled nodes between the YANG-Push publisher
        and the receiver given that the data nodes contained in anydata
        subtree is potentially incomplete (filtered out by xpath or subtree).
        How can a YANG-Push receiver validate the content of anydata nodes?
        <xref target="I-D.aelhassany-anydata-validation"/> addresses this by
        using YANG Library <xref target="RFC8525"/> as mechanism to exchange
        the YANG model of the nodes contained in anydata.</dd>

        <dt>Open Point 2:</dt>

        <dd>The NETCONF Notification structure is defined in <xref
        target="RFC5277"/> using a XSD Schema. For YANG-push <xref
        target="RFC8641"/>, this XSD Schema has been proposed using YANG 1.1
        <xref target="RFC7950"/> modeling language in <xref
        target="I-D.ahuang-netconf-notif-yang"/>. Examples of notifications
        encoded in XML are provided in Section 5 of <xref target="RFC5277"/>.
        In YANG-JSON <xref target="RFC7951"/>, the specification does not
        provide any examples on how notifications should be encoded. In
        YANG-CBOR <xref target="RFC9254"/>, notifications are considered
        "container-like" instances and examples does not show consistency with
        XML-based and YANG-JSON encoding notifications. Assumptions are being
        made in <xref target="I-D.ahuang-netconf-notif-yang"/> providing
        examples of YANG-JSON and YANG-CBOR encoded notifications. The
        notification structure needs consistency accross YANG encodings.
        Confirm findings and propose how to be addressed.</dd>

        <dt>Open Point 3:</dt>

        <dd>Test with running code wherever with <xref
        target="I-D.ietf-netconf-yang-notifications-versioning"/> and <xref
        target="I-D.lincla-netconf-yang-library-augmentation"/> all
        datastore-subtree-filter or datastore-xpath-filter referenced YANG
        modules and their dependencies can be fully indentified.</dd>
      </dl>
    </section>

    <section anchor="Security" title="Security Considerations">
      <t>TBD</t>
    </section>

    <section anchor="Implementation" title="Implementation Status">
      <t>Note to the RFC-Editor: Please remove this section before
      publishing.</t>

      <section anchor="YANG_Schema_Registry_Extension"
               title="YANG Schema Registry Extension">
        <t>Ahmed Elhassany is developing a YANG Schema Extension in Confluent
        Schema Registry.</t>

        <t>The source code can be obtained here: <xref target="YSR24"/>, the
        progress report here: <xref target="YSRPR24"/>, and was validated at
        the IETF 117 hackathon.</t>
      </section>

      <section anchor="YANG-Push_Receiver_Parsing_Library"
               title="YANG-Push Receiver Parsing Library">
        <t>Zhuoyao Lin developed as part of her internship a library to parse
        YANG-Push subscription notifications, identify YANG module
        dependencises with <xref target="RFC8525">YANG Library</xref> and
        obtain with NETCONF &lt;get-schema&gt; rpc calls <xref
        target="RFC6022"/> all YANG modules from YANG-Push publisher.</t>

        <t>The source code can be obtained here: <xref target="LYP23"/> and
        was validated at the IETF 117 hackathon.</t>
      </section>

      <section anchor="YANG-Library_Augmented-by-Addition"
               title="YANG Library Augmented-by Addition">
        <t>Zhuoyao Lin implemented <xref
        target="I-D.lincla-netconf-yang-library-augmentation"/> in order to
        discover augmented-by YANG modules in <xref target="RFC8525">YANG
        Library</xref>.</t>

        <t>The source code can be obtained here: <xref target="YLA24"/> and
        was validated at the IETF 119 hackathon.</t>
      </section>
    </section>

    <section anchor="Acknowledgements" title="Acknowledgements">
      <t>The authors would like to thank Yannick Buchs and Benoit Claise for
      their review and valuable comments. Alex Huang Feng, Jean Quilbeuf and
      Zhuoyao Lin for review and contributing code and providing examples and
      inputs to the open points.</t>
    </section>
  </middle>

  <back>
    <references title="Normative References">
      <?rfc include='reference.RFC.2119'?>

      <?rfc include='reference.RFC.6022'?>

      <?rfc include='reference.RFC.7950'?>

      <?rfc include='reference.RFC.7951'?>

      <?rfc include='reference.RFC.8174'?>

      <?rfc include='reference.RFC.8639'?>

      <?rfc include='reference.RFC.8641'?>

      <?rfc include='reference.RFC.9254'?>

      <?rfc include='reference.I-D.tgraf-netconf-notif-sequencing'?>

      <?rfc include='reference.I-D.tgraf-netconf-yang-push-observation-time'?>

      <?rfc include='reference.I-D.lincla-netconf-yang-library-augmentation'?>

      <?rfc include='reference.I-D.ahuang-netconf-notif-yang'?>

      <?rfc include='reference.I-D.ietf-netconf-yang-notifications-versioning'?>

      <?rfc include='reference.I-D.aelhassany-anydata-validation'?>

      <reference anchor="YSR24"
                 target="https://github.com/confluentinc/schema-registry-yang-format/">
        <front>
          <title>YANG Schema Registry Extension</title>

          <author fullname="Ahmed Elhassany" initials="A." surname="Elhassany"/>

          <date month="February" year="2024"/>
        </front>

        <refcontent>Apache Software Foundation</refcontent>
      </reference>

      <reference anchor="YSRPR24"
                 target="https://github.com/network-analytics/draft-daisy-kafka-yang-integration/blob/main/YANG%20Schema%20registry%20integration.pdf">
        <front>
          <title>YANG Schema Registry Extension Progress Report</title>

          <author fullname="Ahmed Elhassany" initials="A." surname="Elhassany"/>

          <date month="February" year="2024"/>
        </front>
      </reference>

      <reference anchor="LYP23"
                 target="https://github.com/network-analytics/libyangpush/">
        <front>
          <title>libyangpush</title>

          <author fullname="Zhuoyao Lin" initials="Z." surname="Lin"/>

          <date month="September" year="2023"/>
        </front>

        <refcontent>Apache Software Foundation</refcontent>
      </reference>

      <reference anchor="YLA24"
                 target="https://github.com/Zephyre777/draft-lincla-netconf-yang-library-augmentation/">
        <front>
          <title>libyangpush</title>

          <author fullname="Zhuoyao Lin" initials="Z." surname="Lin"/>

          <date month="March" year="2024"/>
        </front>

        <refcontent>Apache Software Foundation</refcontent>
      </reference>

      <reference anchor="Yak24"
                 target="https://github.com/yang-central/yangkit/">
        <front>
          <title>Yangkit</title>

          <author fullname="Frank Feng" initials="F." surname="Feng"/>

          <date month="February" year="2024"/>
        </front>

        <refcontent>Apache Software Foundation</refcontent>
      </reference>

      <reference anchor="Kaf11" target="https://kafka.apache.org/">
        <front>
          <title>Apache Kafka</title>

          <author fullname="Neha Narkhede" initials="N." surname="Narkhede"/>

          <date month="January" year="2011"/>
        </front>

        <refcontent>Apache Software Foundation</refcontent>
      </reference>

      <reference anchor="Con18"
                 target="https://github.com/confluentinc/schema-registry/">
        <front>
          <title>Confluent Schema Registry</title>

          <author fullname="Robert Yokota" initials="R." surname="Yokota"/>

          <date month="December" year="2018"/>
        </front>

        <refcontent>Confluent Community and Apache Software
        Foundation</refcontent>
      </reference>

      <reference anchor="ConDoc18"
                 target="https://docs.confluent.io/platform/current/schema-registry/">
        <front>
          <title>Confluent Schema Registry Documentation</title>

          <author fullname="Robert Yokota " initials="R." surname="Yokota"/>

          <date month="December" year="2018"/>
        </front>

        <refcontent>Confluent Community and Apache Software
        Foundation</refcontent>
      </reference>
    </references>

    <references title="Informative References">
      <?rfc include='reference.RFC.3535'?>

      <?rfc include='reference.RFC.5277'?>

      <?rfc include='reference.RFC.6241'?>

      <?rfc include='reference.RFC.6991'?>

      <?rfc include='reference.RFC.8040'?>

      <?rfc include='reference.RFC.8525'?>

      <?rfc include='reference.RFC.8640'?>

      <?rfc include='reference.RFC.9232'?>

      <?rfc include='reference.I-D.ietf-netconf-https-notif'?>

      <?rfc include='reference.I-D.ietf-netconf-udp-notif'?>

      <?rfc include='reference.I-D.ietf-netconf-distributed-notif'?>

      <reference anchor="W3C.REC-xml-20081126"
                 derivedAnchor="W3C.REC-xml-20081126" quoteTitle="true"
                 target="https://www.w3.org/TR/2008/REC-xml-20081126">
        <front>
          <title>Extensible Markup Language (XML) 1.0 (Fifth Edition)</title>

          <author fullname="Tim Bray" initials="T." surname="Bray">
            <organization showOnFrontPage="true"/>
          </author>

          <author fullname="Jean Paoli" initials="J." surname="Paoli">
            <organization showOnFrontPage="true"/>
          </author>

          <author fullname="Michael Sperberg-McQueen" initials="M."
                  surname="Sperberg-McQueen">
            <organization showOnFrontPage="true"/>
          </author>

          <author fullname="Eve Maler" initials="E." surname="Maler">
            <organization showOnFrontPage="true"/>
          </author>

          <author fullname="Francois Yergeau" initials="F." surname="Yergeau">
            <organization showOnFrontPage="true"/>
          </author>

          <date month="November" year="2008"/>
        </front>

        <refcontent>World Wide Web Consortium Recommendation
        REC-xml-20081126</refcontent>
      </reference>

      <reference anchor="Deh22"
                 target="https://www.oreilly.com/library/view/data-mesh/9781492092384/">
        <front>
          <title>Data Mesh</title>

          <author fullname="Zhamak Dehghani" initials="Z." surname="Dehghani"/>

          <date month="March" year="2022"/>
        </front>

        <seriesInfo name="ISBN" value="9781492092391"/>

        <refcontent>O'Reilly Media</refcontent>
      </reference>

      <reference anchor="Rab07" target="https://rabbitmq.com/">
        <front>
          <title>RabbitMQ</title>

          <author fullname="VMware"/>

          <date month="February" year="2007"/>
        </front>

        <refcontent>Mozilla Public License</refcontent>
      </reference>

      <reference anchor="Atl15" target="https://atlas.apache.org/">
        <front>
          <title>Apache Atlas</title>

          <author fullname="Hortonworks"/>

          <date month="May" year="2015"/>
        </front>

        <refcontent>Apache Software Foundation</refcontent>
      </reference>
    </references>
  </back>
</rfc>