Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

netixlan conflict resolution idea #627

Open
ccaputo opened this issue Jan 9, 2020 · 19 comments
Open

netixlan conflict resolution idea #627

ccaputo opened this issue Jan 9, 2020 · 19 comments
Milestone

Comments

@ccaputo
Copy link
Contributor

@ccaputo ccaputo commented Jan 9, 2020

Capturing elements of what I posted to the data ownership task force and the @peeringdb/pc since there has been support:

With respect to the REST API netixlan data (*: see below for example), I agree there is co-ownership.

I assert that if an IXP is providing an automatic export of their assignments via the IX-F JSON format, that PeeringDB should not present assignments which disagree with the IXP-provided data.

At the same, PeeringDB should only present assignments which have been affirmatively supplied by a network.

Thus, while the DATABASE netixlan table is supplied by the network users, the web site presentation, at least as far as the IXP web pages, and REST API netixlan results should only present data that represents the mutual agreement of the IX-F JSON data and the DATABASE netixlan data.

The current IX-F JSON importer is designed to run from a cron task. Keeping in mind it is currently disabled, as part of what precipitated this task force, when it runs it does the following:

  • fills in data for those networks that have opted to have automatic updates.

  • removes data supplied by networks which is in conflict.

I believe the importer needs to perform an additional task:

  • populate/update a new database table, potentially known as netixlan_ixp, which represents the IXP viewpoint.

and once that is implemented, it can cease removing conflicted assignments unless a network has opted for IX-F JSON automatic updates.

The netixlan_ixp aspect of the importer needs to run frequently, such as once per hour.

Then, for IXPs that have a populated database netixlan_ixp table, the web site IXP pages and REST API netixlan and "peeringdb sync" peeringdb_ixlan table may only present IP assignments that exist in BOTH the netixlan_ixp and netixlan tables.

In this proposal, web site network pages will continue to show netixlan data which is not public in the other views, with an indicator to the viewer that the data is preliminary or not yet sanctioned.

Issue #585 could then be adjusted to only generate tickets for conflicts that are older than a certain amount of time, ideally exceeding the amount of time a conflict may exist as part of normal IXP/network provisioning.

*: So we are all clear on what REST API netixlan is, from my example:

the result is below, in paraphrased form.

{
  "meta": {},
  "data": [
    {
      "id": 1534,
      "net_id": 416,
      "net": { [...] },
      "ix_id": 13,
      "name": "SIX Seattle: MTU 1500",
      "ixlan_id": 13,
      "ixlan": { [...] },
      "notes": "",
      "speed": 20000,
      "asn": 6456,
      "ipaddr4": "206.81.80.10",
      "ipaddr6": "2001:504:16::1938",
      "is_rs_peer": true,
      "created": "2010-07-29T00:00:00Z",
      "updated": "2019-03-02T03:15:13Z",
      "status": "ok"
    }
  ]
}

Also, from my example:

in paraphrased form:

{
  "meta": {},
  "data": [
    {
      "id": 13,
      "ix_id": 13,
      "ix": { [...] },
      "name": "MTU 1500",
      "descr": "",
      "mtu": 1500,
      "dot1q_support": false,
      "rs_asn": 0,
      "arp_sponge": null,
      "net_set": [ FULL LIST OF PARTICIPANTS FROM WHICH IPs CAN BE DERIVED ],
      "ixpfx_set": [ ... ],
      "created": "2010-07-29T00:00:00Z",
      "updated": "2019-05-08T03:57:10Z",
      "status": "ok"
    }
  ]
}
@arnoldnipper

This comment has been minimized.

Copy link
Contributor

@arnoldnipper arnoldnipper commented Jan 9, 2020

Then, for IXPs that have a populated database netixlan_ixp table, the web site IXP pages and REST API netixlan and "peeringdb sync" peeringdb_ixlan table may only present IP assignments that exist in BOTH the netixlan_ixp and netixlan tables.

Effectively that would mean deleting entries where net and ix disagree. Right? #585 however exactly fixes this.

@ccaputo

This comment has been minimized.

Copy link
Contributor Author

@ccaputo ccaputo commented Jan 9, 2020

Then, for IXPs that have a populated database netixlan_ixp table, the web site IXP pages and REST API netixlan and "peeringdb sync" peeringdb_ixlan table may only present IP assignments that exist in BOTH the netixlan_ixp and netixlan tables.

Effectively that would mean deleting entries where net and ix disagree. Right? #585 however exactly fixes this.

No. The network's netixlan entry would remain untouched and when presented in a viewable fashion it could be denoted as being in conflict with authoritative data, so that viewers know to consider it suspect or premature and so the network itself is aware of the problem in case they miss any notifications due to #585 .

@ccaputo

This comment has been minimized.

Copy link
Contributor Author

@ccaputo ccaputo commented Jan 9, 2020

This will also enable us to have a feature of being able to indicate to a logged-in admin for a network when their supplied data is incomplete, and they could either be prompted to auto-add individual missing assignments and/or encouraged to turn on the auto-updater.

@arnoldnipper

This comment has been minimized.

Copy link
Contributor

@arnoldnipper arnoldnipper commented Jan 9, 2020

You write: the web site IXP pages and REST API netixlan and "peeringdb sync" peeringdb_ixlan table may only present IP assignments that exist in BOTH the netixlan_ixp and netixlan tables.

Still, if the ix doesn't have an entry this is equivalent to a removal. Right?

@ccaputo

This comment has been minimized.

Copy link
Contributor Author

@ccaputo ccaputo commented Jan 9, 2020

You write: the web site IXP pages and REST API netixlan and "peeringdb sync" peeringdb_ixlan table may only present IP assignments that exist in BOTH the netixlan_ixp and netixlan tables.

Still, if the ix doesn't have an entry this is equivalent to a removal. Right?

No. The whole proposal, including the "BOTH" aspect, is only for "if an IXP is providing an automatic export of their assignments via the IX-F JSON format".

In the case where no IXP-provided data is present, or if the timestamp of the last provision of IXP data is stale, there is no conflict and so the network-provided data can be presented. Stale to be determined, but I would suggest on the order of 30 days.

@grizz

This comment has been minimized.

Copy link
Member

@grizz grizz commented Jan 9, 2020

+1 from me, this is a very elegant solution to the importer problem and allows us to retain data from the IX-F import while not disrupting data a network has entered until they ask us too.

@arnoldnipper

This comment has been minimized.

Copy link
Contributor

@arnoldnipper arnoldnipper commented Jan 10, 2020

In the case where no IXP-provided data is present, or if the timestamp of the last provision of IXP data is stale, there is no conflict and so the network-provided data can be presented. Stale to be determined, but I would suggest on the order of 30 days.

If the IX has removed the net from its network then there is no IX-F JSON entry. But the network-presented data might still be there. This clearly is a conflict. No?

@ccaputo

This comment has been minimized.

Copy link
Contributor Author

@ccaputo ccaputo commented Jan 10, 2020

In the case where no IXP-provided data is present, or if the timestamp of the last provision of IXP data is stale, there is no conflict and so the network-provided data can be presented. Stale to be determined, but I would suggest on the order of 30 days.

If the IX has removed the net from its network then there is no IX-F JSON entry. But the network-presented data might still be there. This clearly is a conflict. No?

This is handled by the above. If an IXP is still providing IX-F JSON records but has removed an IP assignment from its export, and the network still has the assignment, the "BOTH" requirement will fail and the assignment will not be presented via the REST API nor /ix/ page.

In pseudo-code:

if (IXP_provides_any_IX-F_JSON_data_for_a_LAN_it_controls within last TBD days)
  {
    if (Network_provides_IP_assignment)
      {
        if (IXP_provides_matching_IP_assignment)
          {
            // No conflict...

            Present the IP assignment via REST API and /ix/ page.

            Present the IP assignment on network's /net/ page.
          }
        else
          {
            // Conflict...

            Do not present the IP assignment via REST API nor /ix/ page.

            Since the network maintains their own /net/ page, present the
            IP assignment on the network's /net/ page, but gracefully
            highlight the conflict in such a way that the network admins
            and any visitors know it is in conflict with IXP.

            OPTIONAL: Once the conflict has existed for TBD days, invoke
            the notification/resolution method described in #585.
          }
      }
    else
      {
        // NOT Network_provides_IP_assignment

        if (IXP_provides_IP_assignment_for_this_ASN)
          {
            OPTIONAL: Present message only to network's admin users when
            they view their /net/ page indicating an IP assignment they
            are missing which they could add or dismiss.  Remind them of
            the IX-F auto-import option.

            Don't show anything in REST API or /ix/ page.
          }
        else
          {
            Show nothing anywhere.
          }
      }
  }
else
  {
    // NOT IXP_provides_any_IX-F_JSON_data_for_a_LAN_it_controls or it is
    // older than TBD days and thus stale...

    if (Network_provides_IP_assignment)
      {
        Present the IP assignment in REST API and /ix/ and /net/ pages.
      }
  }
@grizz grizz added this to the 1 Decide milestone Jan 10, 2020
@fkorsback

This comment has been minimized.

Copy link

@fkorsback fkorsback commented Jan 14, 2020

+1 from me here aswell, i think this should cover all of the possible cases where a conflict might occur. good to go

@listerr

This comment has been minimized.

Copy link

@listerr listerr commented Jan 23, 2020

Possibly attempt to notify an IXP that their IX-F data is failing to load?

@arnoldnipper

This comment has been minimized.

Copy link
Contributor

@arnoldnipper arnoldnipper commented Jan 23, 2020

Possibly attempt to notify an IXP that their IX-F data is failing to load?

@listerr the idea is to notify each party involved (net, ix) when an IX-F import will lead to conflicts

@ccaputo

This comment has been minimized.

Copy link
Contributor Author

@ccaputo ccaputo commented Jan 24, 2020

Clarifying REST API viewability, based on discussions:

With this proposal, conflicted data won't be visible in the /net/ result or other results as long as it is conflicted. The idea here is to prevent PeeringDB from presenting (via web or API) any data that could be interpreted as valid when it is not valid. Conflicted data is not valid. While that conflict can be denoted in the web pages, so a network has a clue that an issue has arisen, for API queries conflicted data would be withheld to prevent accidental automation using conflicted data.

For API developers the lack of consistency between a GET after a PUT/POST is the way a conflict is indicated.

@ccaputo

This comment has been minimized.

Copy link
Contributor Author

@ccaputo ccaputo commented Jan 24, 2020

From https://lists.peeringdb.com/pipermail/dataownership-tf/2020-January/000094.html :

My sense of of what happened is as follows:

    1. IXP gave Network an IP assignment prior to the assignment being in the IXP's IX-F JSON export.
    1. Network added IP assignment to their PeeringDB page.
    1. IX-F importer run at 0000 UTC removed the IP assignment added by the Network since the IXPs IX-F JSON export did not have a corresponding record.
    1. Network noticed the IP assignment had gone missing and re-added the IP assignment, since surely IXP will update their IX-F JSON export soon.
    1. Next day at 0000 UTC, IX-F importer ran again, and data was removed again.
    1. Process repeats and frustration ensues, with the Network having no idea when it won't be a waste of effort to re-add the IP assignment, unless they scrutinize the IXPs IX-F JSON export data.
      Not scalable and a pain in the neck.

What #627 does is change the above to be:

    1. IXP gives Network an IP assignment prior to the assignment being in the IXPs IX-F JSON export.
    1. Network adds IP assignment to their PeeringDB page.
    1. PeeringDB considers the Network's IP addition in the context of whether the IXP regularly provides IX-F JSON exports. This is where the big if-then-else statement at #627 (comment) is evaluated, the gist being:

      • 3a. If IXP does export JSON and the data is not in conflict, the data is presented as normal.

      • 3b. If IXP does export JSON and the data is in conflict, the data is flagged as being in conflict. No presentation via REST API or /ix/ page, while data is presented on Network's /net/ page with a conflict indicator.

      • 3c. If IXP does not export JSON, simply present the Network provided data.

    1. As time passes, if/when the IX-F JSON export data has a matching IP assignment, any conflict is automatically resolved and the data is presented as normal.

Thus #627 avoids the frustration of a network needing to add and re-add data until the IXP providing an IX-F JSON export contains matching data. The network still owns their data - it hasn't been removed from their web page - and the IX-F JSON exporting IXP still owns their subnet, controlling whether provisioning kicks off. (This also prevents unnecessary broadcasts/multicasts on peering fabrics, due to premature provisioning.)

@arnoldnipper

This comment has been minimized.

Copy link
Contributor

@arnoldnipper arnoldnipper commented Jan 25, 2020

With this proposal, conflicted data won't be visible in the /net/ result or other results as long as it is conflicted. The idea here is to prevent PeeringDB from presenting (via web or API) any data that could be interpreted as valid when it is not valid. Conflicted data is not valid.

I totally disagree with the sentence "Conflicted data is not valid". In case of a conflict, the conflict has to be resolved asap. Until then no data must be changed.

@ccaputo

This comment has been minimized.

Copy link
Contributor Author

@ccaputo ccaputo commented Jan 25, 2020

With this proposal, conflicted data won't be visible in the /net/ result or other results as long as it is conflicted. The idea here is to prevent PeeringDB from presenting (via web or API) any data that could be interpreted as valid when it is not valid. Conflicted data is not valid.

I totally disagree with the sentence "Conflicted data is not valid". In case of a conflict, the conflict has to be resolved asap. Until then no data must be changed.

When the IX-F importer was running, it did exactly that!

Upon initial implementation of #627, conflicted data would disappear from all views except the /net/ (aka /asn/) HTML page. On the /net/ (aka /asn/) HTML page it would be flagged as conflicted.

After initial run of #627, no data would disappear, except in cases where either the IXP or the Network no longer lists an assignment. That is intended, since the idea is that mutual agreement is needed in order for the assignment to be public.

In the same way that no one is allowed to force data onto a Network's PeeringDB pages, no one should be allowed to force data onto an IXP's PeeringDB pages if an IXP makes it clear they do not want that. As I understand things, that's always been a fundamental PeeringDB concept with respect to Network pages, and I don't see why that shouldn't also apply to IXP pages.

Including some other stuff I wrote in a task-force email:

Either data from the IXP is authoritative or it is not.

If an IXP's IX-F JSON export is properly formatted and decoded, I believe it should be considered authoritative, giving the IXP the ability to prevent conflicted data from being propagated and used to inform provisioning systems.

Propagation of conflicted data affects peering fabrics by either causing broadcast/multicast storms for missing routers or by having networks attempt connections to routers using incorrect ASNs.

IXP data might contain errors, as may PeeringDB data. That has always been the case. Further, both sources of data can be hacked, and the value of PeeringDB as a hacker target only goes up if networks are automating BGP sessions in a senseless manner. Thus networks that use data provided by PeeringDB should be careful in their automation, such as by involving human review, or handling changes in source data in a manner resistant to unintended disruption.

and:

Network-provided data stays intact in the netixlan table, but it is disputable (hence this discussion) that it should be public in such as way as to inform provisioning. I believe the IXP should have the option of having a say in that matter, since it is their address space, and their fabric which can be affected.

@arnoldnipper

This comment has been minimized.

Copy link
Contributor

@arnoldnipper arnoldnipper commented Jan 25, 2020

Network-provided data stays intact in the netixlan table, but it is disputable (hence this discussion) that it should be public in such as way as to inform provisioning. I believe the IXP should have the option of having a say in that matter, since it is their address space, and their fabric which can be affected.

IMO we are running in circles somehow.

We want to have the IX-F importer, but also want to make it safe in the sense that data is not deleted/gets invisible in case of an error on either side.

As we are not able to resolve a conflict automatically w/o causing interruption of services, the conflict has to be brought to attention immediately by notifying each party involved and then get resolved.

AC will do this resolution. And I fully agree that the IXP should be the authoritative source of information regarding (asn, ipv4, ipv6).

If there is anything we can do to minimize the work of AC w/o violating above mentioned principles, we should implement it.

@ccaputo

This comment has been minimized.

Copy link
Contributor Author

@ccaputo ccaputo commented Jan 25, 2020

Network-provided data stays intact in the netixlan table, but it is disputable (hence this discussion) that it should be public in such as way as to inform provisioning. I believe the IXP should have the option of having a say in that matter, since it is their address space, and their fabric which can be affected.

IMO we are running in circles somehow.

We want to have the IX-F importer, but also want to make it safe in the sense that data is not deleted/gets invisible in case of an error on either side.

As we are not able to resolve a conflict automatically w/o causing interruption of services, the conflict has to be brought to attention immediately by notifying each party involved and then get resolved.

AC will do this resolution. And I fully agree that the IXP should be the authoritative source of information regarding (asn, ipv4, ipv6).

If there is anything we can do to minimize the work of AC w/o violating above mentioned principles, we should implement it.

Hi Arnold,

Maybe the way to reach consensus here is to revise part of #627's initial
proposal:

  • "Issue #585 could then be adjusted to only generate tickets for
    conflicts that are older than a certain amount of time, ideally
    exceeding the amount of time a conflict may exist as part of normal
    IXP/network provisioning."

to rather be:

  • "At the same time as #627 (netixlan conflict resolution idea)
    withholds general API and /ix/ HTML page publication of conflicted
    data to avoid premature and/or accidental provisioning, and produces a
    conflict notification on the Network's /net/ HTML page, the resolution
    methods of #585 (Interim IX-F JSON importer) are also triggered.
    This enables the Admin Committee to exercise discretion and/or brings
    parties together toward agreement, resulting in either a correction or
    override of the IX-F JSON export from the IXP and/or a change in the
    data provided by the Network, as needed to aide in resolution of the
    conflict. To facilitate an override, a netixlan_override table is
    envisioned, to accompany the netixlan table and new netixlan_ixp
    table. The Admin Committee also has the discretion to disable the
    IXP's IX-F JSON import if it is apparent new data is in error, thus
    minimizing the duration of accidental censoring of valid data."

? Does that work for you?

Then after implementation and experience is gained, if the trigger delay
for invoking #585 turns out to be too low at zero seconds, it could be
increased over time, such as to an hour or more, if it makes sense to do
so in the judgment of the Admin Committee.

@wmarantz

This comment has been minimized.

Copy link

@wmarantz wmarantz commented Jan 27, 2020

@fkorsback would you mind providing more insight on how you consume the peeringDB data? As I read things, for this proposal to help the delete issue an operator would either need to consume the data by database sync or from a network web page ( not via API or via an IX page ). Or, the consumption method could not be the underlying issue and you are tried of reentering data you input that IX-F import deleted.

I'm in favor of immediate ( per agreed IX-F import frequency ) notification of conflict on both the web page and the email/ticket system.

I started looking at this issue from the view of a network operator trying to accelerate peering at new IXPs while dealing with unclear internal provisioning intervals for IX interfaces and peers. I was originally hoping the conflicted data would be available via the API in some way that a consumer could know that they are pulling potentially conflicted data and deconflict independently if desired. However, after hearing the view of IXP operators on broadcast storms I support limiting the API and netixlan web page visibility until IXP and user supplied data are in agreement.

I don't support data deletes unless the network has agreed to let the IXP fully manage their entries.

@arnoldnipper

This comment has been minimized.

Copy link
Contributor

@arnoldnipper arnoldnipper commented Jan 27, 2020

I don't support data deletes unless the network has agreed to let the IXP fully manage their entries.

+1 ... to be crystal clear. I don't support automatic data deletes ... and for early provisioning #539 should help. This feature is ready for implementation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants
You can’t perform that action at this time.