Skip to content

Commit

Permalink
Allow excluding certain types in payload output. (#866)
Browse files Browse the repository at this point in the history 
This PR add support for excluding certain payload types from output both in
the vrps command and the HTTP server. With this in place, the PR also adds
router key and ASPA to the standard json output format.

For the vrps command, new command line options --no-route-origins,
--no-router-keys, and --no-aspas have been added. For the payload output
endpoints of the HTTP server, a new excludes query parameter can be used to
exclude routeOrigins, routerKeys, or aspas.

These flags are available even for formats that don’t actually support
outputting all these payload types.

When router keys or aspa has not been disabled, those payload types are now
properly excluded rather than adding empty sections for them.

---------

Co-authored-by: Alex Band <alex@nlnetlabs.nl>
Co-authored-by: Luuk Hendriks <luuk@nlnetlabs.nl>
  • Loading branch information
@partim @AlexanderBand @DRiKE
3 people committed May 31, 2023
1 parent 1955937 commit e1b3738
Show file tree
Hide file tree
Showing 12 changed files with 830 additions and 419 deletions.
4 changes: 2 additions & 2 deletions doc/manual/source/advanced-features.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ You can let Routinator process ASPA objects and include them in the published
dataset, as well as the metrics, using the :option:`--enable-aspa` option
or by setting ``enable-aspa`` to True in the :doc:`configuration
file<configuration>`. ASPA information will be exposed via RTR, as well as
in the :term:`jsonext` output format, e.g.:
in the :term:`json` and :term:`jsonext` output formats, e.g.:

.. code-block:: json
Expand Down Expand Up @@ -96,7 +96,7 @@ You can let Routinator process router keys and include them in the published
dataset, as well as the metrics, using the :option:`--enable-bgpsec` option
or by setting ``enable-bgpsec`` to True in the :doc:`configuration
file<configuration>`. BGPsec information will be exposed via RTR, as well as
in the :term:`SLURM` and :term:`jsonext` output format, e.g.:
in the :term:`SLURM`, :term:`json` and :term:`jsonext` output formats, e.g.:

.. code-block:: json
Expand Down
9 changes: 9 additions & 0 deletions doc/manual/source/http-service.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,15 @@ This will produce the following output:
]
}
The query parameter ``exclude`` can be used to exclude certain payload types
from the response. The values ``routeOrigins``, ``routerKeys``, and ``aspas``
disable inclusion of route origins, router keys, and ASPAs, respectively. The
values can either be given in separate ``exclude`` parameters or included in
one separated by commas.

.. versionadded:: 0.13.0
Allow excluding specific data from the output

More Specific Prefixes
""""""""""""""""""""""

Expand Down
21 changes: 21 additions & 0 deletions doc/manual/source/interactive.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,15 @@ error. You can influence the amount of details returned with the
:option:`--verbose` and :option:`--quiet` options. To learn more about what kind
of information returned, refer to the :doc:`logging` section.

If you have enabled :ref:`advanced-features:bgpsec` and/or
:ref:`advanced-features:aspa` validation, in some output formats the amount
of data can be quite overwhelming. You can exclude specific data types for the
output with the :option:`--no-route-origins`, :option:`--no-router-keys` and
the :option:`--noaspas` options.

.. versionchanged:: 0.13.0
Allow excluding specific data from the output.

Query Options
-------------

Expand Down Expand Up @@ -128,7 +137,19 @@ You will now see that a more specific /23 prefix is returned as well:
will get a list of all more specific VRPs covered by the prefix you
supplied in the query.

Exclude Specific Data Types
"""""""""""""""""""""""""""

If you have enabled :ref:`advanced-features:bgpsec` and/or
:ref:`advanced-features:aspa` validation, in some output formats the amount of
data can be quite overwhelming. You can exclude specific payload types with
the :option:`--no-route-origins`, :option:`--no-router-keys` and
:option:`--noaspas` options to disable inclusion of route origins, router
keys, and ASPAs, respectively.

.. deprecated:: 0.9.0
``--filter-asn`` and ``--filter-prefix``
.. versionchanged:: 0.11.0
Add the :option:`--more-specifics` option
.. versionadded:: 0.13.0
Allow excluding specific data from the output
69 changes: 52 additions & 17 deletions doc/manual/source/manual-page.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -458,18 +458,35 @@ These can be requested by providing different commands on the command line.
be in one single output file.

json
The list is placed into a JSON object with a single element
*roas* which contains an array of objects with four
The list is placed into a JSON object with up to four
members: *roas* contains the validated route origin
authorizations, *routerKeys* contains the validated
BGPsec router keys, *aspas* contains the validated
ASPA payload, and *metadata* contains some information
about the validation run itself. Of the first three, only
those members are present that have not been disabled or
excluded.

The *roas* member contains an array of objects with four
elements each: The autonomous system number of the network
authorized to originate a prefix in *asn*, the prefix in
slash notation in *prefix*, the maximum prefix length of
the announced route in *maxLength*, and the trust anchor
from which the authorization was derived in *ta*. This
format is identical to that produced by the RIPE NCC RPKI
Validator except for different naming of the trust anchor.
Routinator uses the name of the TAL file without the
extension *.tal* whereas the RIPE NCC Validator has a
dedicated name for each.
from which the authorization was derived in *ta*.

The *routerKeys* member contains an array of objects with
four elements each: The autonomous system using the router
key is given in *asn*, the key identifier as a string of
hexadecimal digits in *SKI*, the actual public key as a
Base 64 encoded string in *routerPublicKey*, and the trust
anchor from which the authorization was derived in *ta*.

The *aspa* member contains an array of objects with four
members each: The *customer* member contains the customer
ASN, *afi* the address family as either "ipv4" or "ipv6",
*providers* contains the provider ASN set as an array, and
the trust anchor from which the authorization was derived
in *ta*.

The output object also includes a member named *metadata*
which provides additional information. Currently, this is a
Expand All @@ -478,16 +495,23 @@ These can be requested by providing different commands on the command line.
which provides the same time but in the standard ISO date
format.

If only route origins are included, this format is identical
to that produced by the RIPE NCC
RPKI Validator except for different naming of the trust
anchor.
Routinator uses the name of the TAL file without the
extension *.tal* whereas the RIPE NCC Validator has a
dedicated name for each.

jsonext
The list is placed into a JSON object with three members:
*roas* contains the validated route origin
The list is placed into a JSON object with up to four
members: *roas* contains the validated route origin
authorizations, *routerKeys* contains the validated
BGPsec router keys, and *metadata* contains some information
about the validation run itself.

All three members are always present, even if BGPsec has
not been enabled. In this case, *routerKeys* will simply
be empty.
BGPsec router keys, *aspas* contains the validated
ASPA payload, and *metadata* contains some information
about the validation run itself. Of the first three, only
those members are present that have not been disabled or
excluded.

The *roas* member contains an array of objects with four
elements each: The autonomous system number of the network
Expand Down Expand Up @@ -618,7 +642,12 @@ These can be requested by providing different commands on the command line.
Note that VRPs with more specific prefixes have no influence on
whether a route is RPKI valid or invalid and therefore these VRPs
are of an informational nature only.


.. option:: --no-route-origins, --no-router-keys, --no-aspas

These three options can be used to exclude the various payload
types from being included in the output.


.. subcmd:: validate

Expand Down Expand Up @@ -1401,6 +1430,12 @@ In addition, the query parameter ``include=more-specifics`` will cause the
inclusion of VRPs for more specific prefixes of prefixes given via
``select-prefix``.

Finally, the query parameter ``exclude`` can be used to exclude certain
payload types from the response. The values ``routeOrigins``, ``routerKeys``,
and ``aspas`` disable inclusion of route origins, router keys, and ASPAs,
respectively. The values can either be given in separate ``exclude``
parameters or included in one separated by commas.

These parameters work in the same way as the options of the same name to the
:subcmd:`vrps` command.

Expand Down
130 changes: 91 additions & 39 deletions doc/manual/source/output-formats.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -68,9 +68,21 @@ generated in a wide range of output formats for various use cases.
rsync://rpki.ripe.net/repository/DEFAULT/73/fe2d72-c2dd-46c1-9429-e66369649411/1/49sMtcwyAuAW2lVDSQBGhOHd9og.roa,AS196615,93.175.147.0/24,24,2021-05-03 14:51:30,2022-07-01 00:00:00
json
The output is in JSON format. The list is placed into a member
named *roas* which contains an array of objects with four elements
each:
The list is placed into a JSON object with up to four members:

- *roas* contains the validated route origin authorisations,
- *routerKeys* contains the validated
:ref:`advanced-features:bgpsec` router keys,
- *aspas* contains the validated :ref:`advanced-features:aspa`
payload, and
- *metadata* contains some information about the validation run
itself.

Of the first three, only those members are present that have not
been disabled or excluded.

The *roas* member contains an array of objects with four elements
each:

- *asn* lists the Autonomous System Number of the network
authorised to originate a prefix,
Expand All @@ -80,9 +92,25 @@ generated in a wide range of output formats for various use cases.
- *ta* has the trust anchor from which the authorisation was
derived.

This format of the *roas* element is identical to that produced by
the RIPE NCC RPKI Validator except for different naming of the
trust anchor.
The *routerKeys* member contains an array of objects with four
elements each:

- *asn* contains the autonomous system using the router key,
- *SKI* lists the key identifier as a string of hexadecimal
digits,
- *routerPublicKey* contains the actual public key as a Base 64
encoded string, and
- *ta* has the trust anchor from which the authorisation was
derived.

The *aspa* member contains an array of objects with four members
each:

- *customer* contains the customer ASN,
- *afi* lists the address family as either "ipv4" or "ipv6",
- *providers* contains the provider ASN set as an array, and
- *ta* has the trust anchor from which the authorisation was
derived.

The output object also includes a member named *metadata* which
provides additional information. Currently, this is a member
Expand All @@ -94,21 +122,37 @@ generated in a wide range of output formats for various use cases.
{
"metadata": {
"generated": 1626853335,
"generatedTime": "2021-07-21T07:42:15Z"
"generated": 1685455841,
"generatedTime": "2023-05-30T14:10:41Z"
},
"roas": [
{ "asn": "AS196615", "prefix": "2001:7fb:fd03::/48", "maxLength": 48, "ta": "ripe" },
{ "asn": "AS196615", "prefix": "2001:7fb:fd04::/48", "maxLength": 48, "ta": "ripe" },
{ "asn": "AS196615", "prefix": "93.175.147.0/24", "maxLength": 24, "ta": "ripe" }
]
"roas": [{
"asn": "AS196615",
"prefix": "93.175.147.0/24",
"maxLength": 24,
"ta": "ripe"
}
],
"routerKeys": [{
"asn": "AS211321",
"SKI": "17316903F0671229E8808BA8E8AB0105FA915A07",
"routerPublicKey": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAET10FMBxP6P3r6aG_ICpfsktp7X6ylJIY8Kye6zkQhNOt0y-cRzYngH8MGzY3cXNvZ64z4CpZ22gf4teybGq8ow",
"ta": "ripe"
}],
"aspas": [{
"customer": "AS64496",
"afi": "ipv6",
"providers": ["AS64499", "AS64511", "AS65551"],
"ta": "ripe"
}]
}
.. versionchanged:: 0.10.0
Add the *metadata* member
.. versionchanged:: 0.13.0
Add the *routerKeys* and *aspas* members

jsonext
The list is placed into a JSON object with four members:
The list is placed into a JSON object with up to four members:

- *roas* contains the validated route origin authorisations,
- *routerKeys* contains the validated
Expand All @@ -118,10 +162,8 @@ generated in a wide range of output formats for various use cases.
- *metadata* contains some information about the validation run
itself.

All four members are always present, even if
:ref:`advanced-features:bgpsec` and/or
:ref:`advanced-features:aspa` have not been enabled. In this case,
*routerKeys* and *aspas* will simply be empty.
Of the first three, only those members are present that have not
been disabled or excluded.

The *roas* member contains an array of objects with four elements
each:
Expand Down Expand Up @@ -252,6 +294,8 @@ generated in a wide range of output formats for various use cases.
Add :ref:`advanced-features:bgpsec` information
.. versionchanged:: 0.13.0
Add :ref:`advanced-features:aspa` information
.. versionchanged:: 0.13.0
Only include members that have not been disabled or excluded

slurm
The list is formatted as locally added assertions of a :doc:`local
Expand Down Expand Up @@ -351,45 +395,53 @@ generated in a wide range of output formats for various use cases.
provide numbers for the complete repository.

For each trust anchor, it will print the number of verified ROAs
and VRPs, as well as router certificates and keys. Note that router
keys will only be verified and included in the totals if you have
enabled :ref:`advanced-features:bgpsec`.
and VRPs, router certificates and keys, as well as ASPAs. Note that
router keys and ASPAs will only be included in the totals if you
have enabled :ref:`advanced-features:bgpsec` and
:ref:`advanced-features:aspa`, respectively.

.. code-block:: text
Summary at 2022-01-28 08:37:27.046365 UTC
Summary at 2023-05-30 16:22:27.060940 UTC
afrinic:
ROAs: 3587 verified;
VRPs: 4545 verified, 3 unsafe, 4466 final;
ROAs: 4896 verified;
VRPs: 6248 verified, 5956 final;
router certs: 0 verified;
router keys: 0 verified, 0 final.
ASPAs: 0 verified, 0 final.
apnic:
ROAs: 18612 verified;
VRPs: 85992 verified, 0 unsafe, 85711 final;
ROAs: 25231 verified;
VRPs: 109978 verified, 109717 final;
router certs: 0 verified;
router keys: 0 verified, 0 final.
ASPAs: 2 verified, 2 final.
arin:
ROAs: 41500 verified;
VRPs: 50495 verified, 5 unsafe, 1812 final;
router certs: 0 verified;
router keys: 0 verified, 0 final.
ROAs: 63188 verified;
VRPs: 78064 verified, 76941 final;
router certs: 1 verified;
router keys: 1 verified, 1 final.
ASPAs: 7 verified, 7 final.
lacnic:
ROAs: 11744 verified;
VRPs: 23628 verified, 0 unsafe, 21235 final;
ROAs: 18036 verified;
VRPs: 32565 verified, 30929 final;
router certs: 0 verified;
router keys: 0 verified, 0 final.
ASPAs: 0 verified, 0 final.
ripe:
ROAs: 27195 verified;
VRPs: 149164 verified, 17 unsafe, 149162 final;
ROAs: 39081 verified;
VRPs: 211048 verified, 211043 final;
router certs: 2 verified;
router keys: 2 verified, 2 final.
ASPAs: 57 verified, 57 final.
total:
ROAs: 141922 verified;
VRPs: 361536 verified, 25 unsafe, 307434 final;
router certs: 2 verified;
router keys: 2 verified, 2 final.
ROAs: 150432 verified;
VRPs: 437903 verified, 434586 final;
router certs: 3 verified;
router keys: 3 verified, 3 final.
ASPAs: 66 verified, 66 final.
.. versionchanged:: 0.11.0
Reformat, sort alphabetically and add
:ref:`advanced-features:bgpsec` information
.. versionadded:: 0.13.0
Include :ref:`advanced-features:aspa`

0 comments on commit e1b3738

Please sign in to comment.