Loading the GUI for prefixes takes more than 60s

[sirtux]

Environment

• Python version: 3.9
• Nautobot version: 1.1.3

Steps to Reproduce

1. Have a nautobot installation with more than 100k prefixes
2. Open https://$nautobot/ipam/prefixes/

Expected Behavior

A snappy respone

Observed Behavior

Load time is more than 60s

Possible workaround

As it seems now, the default is to load all prefixes in all VRFs etc in the background,
but only render the first ones.

Either the query in the backend has to be limited to only load what will be shown,
or:
When accessing the prefix page, nothing is loaded until a specific search criteria is entered.
This criteria can be saved as a default per user

[glennmatthews]

Thanks for the report! What "per page" setting are you using for this table view? Decreasing it may help in the interim while we investigate this issue.

(I've done some initial profiling of this issue with the Django Debug Toolbar and it doesn't appear that the database queries are the slowest part - something in the page rendering logic instead, perhaps?)

[sirtux]

It's set to 500. I tried reducing it to 250, and it then only takes about 13s - which is much better, but still too long in my opinion...

[glennmatthews]

Probably the way this will need to be addressed will be with some data model changes - actually representing parent/child prefixes as a tree in the database, using common table expressions, something along those lines.

This is somewhat related to our use of django-mptt in other models, and the need to replace it throughout (#510).

[lampwins]

This has also become a show stopper for me lately. I have been playing with a new setting DISABLE_PREFIX_LIST_HIERARCHY which as the name implies will not apply the parent/child annotations and thus list all prefixes in the list view as a flat set. I feel this is an amicable workaround for affected users until we commit to a larger data model overhaul which would realistically present breaking changes, and thus be further out.

[lampwins]

(I've done some initial profiling of this issue with the Django Debug Toolbar and it doesn't appear that the database queries are the slowest part - something in the page rendering logic instead, perhaps?)

It is positively the query in the annotate_tree() method, as profiled here:

SELECT "ipam_prefix"."id",
       "ipam_prefix"."created",
       "ipam_prefix"."last_updated",
       "ipam_prefix"."_custom_field_data",
       "ipam_prefix"."status_id",
       "ipam_prefix"."network",
       "ipam_prefix"."broadcast",
       "ipam_prefix"."prefix_length",
       "ipam_prefix"."site_id",
       "ipam_prefix"."vrf_id",
       "ipam_prefix"."tenant_id",
       "ipam_prefix"."vlan_id",
       "ipam_prefix"."role_id",
       "ipam_prefix"."is_pool",
       "ipam_prefix"."description",

  (SELECT COUNT(*) AS "count"
   FROM "ipam_prefix" U0
   WHERE (U0."prefix_length" < "ipam_prefix"."prefix_length"
          AND U0."network" <= "ipam_prefix"."network"
          AND U0."broadcast" >= "ipam_prefix"."broadcast"
          AND COALESCE(U0."vrf_id", 3def19a1-48cb-4fa9-a98a-5e48a3c24cc0) = COALESCE("ipam_prefix"."vrf_id", 3def19a1-48cb-4fa9-a98a-5e48a3c24cc0))
   LIMIT 1) AS "parents",

  (SELECT COUNT(*) AS "count"
   FROM "ipam_prefix" U0
   WHERE (U0."prefix_length" > "ipam_prefix"."prefix_length"
          AND U0."network" >= "ipam_prefix"."network"
          AND U0."broadcast" <= "ipam_prefix"."broadcast"
          AND COALESCE(U0."vrf_id", 3def19a1-48cb-4fa9-a98a-5e48a3c24cc0) = COALESCE("ipam_prefix"."vrf_id", 3def19a1-48cb-4fa9-a98a-5e48a3c24cc0))
   LIMIT 1) AS "children"
FROM "ipam_prefix"
LEFT OUTER JOIN "ipam_vrf" ON ("ipam_prefix"."vrf_id" = "ipam_vrf"."id")
ORDER BY "ipam_vrf"."name" ASC NULLS FIRST,
                               "ipam_prefix"."network" ASC,
                               "ipam_prefix"."prefix_length" ASC
LIMIT 50
OFFSET 29700

Query PlanLimit  (cost=46598305.44..46676740.21 rows=50 width=379)
  ->  Result  (cost=8049.09..47072051.48 rows=30002 width=379)
        ->  Sort  (cost=8049.09..8124.09 rows=30002 width=363)
              Sort Key: ipam_vrf.name NULLS FIRST, ipam_prefix.network, ipam_prefix.prefix_length
              ->  Hash Left Join  (cost=12.25..792.03 rows=30002 width=363)
                    Hash Cond: (ipam_prefix.vrf_id = ipam_vrf.id)
                    ->  Seq Scan on ipam_prefix  (cost=0.00..701.02 rows=30002 width=145)
                    ->  Hash  (cost=11.00..11.00 rows=100 width=234)
                          ->  Seq Scan on ipam_vrf  (cost=0.00..11.00 rows=100 width=234)
        SubPlan 1
          ->  Limit  (cost=784.33..784.34 rows=1 width=8)
                ->  Aggregate  (cost=784.33..784.34 rows=1 width=8)
                      ->  Bitmap Heap Scan on ipam_prefix u0  (cost=183.30..784.32 rows=6 width=0)
                            Recheck Cond: (prefix_length < ipam_prefix.prefix_length)
                            Filter: ((network <= ipam_prefix.network) AND (broadcast >= ipam_prefix.broadcast) AND (COALESCE(vrf_id, '3def19a1-48cb-4fa9-a98a-5e48a3c24cc0'::uuid) = COALESCE(ipam_prefix.vrf_id, '3def19a1-48cb-4fa9-a98a-5e48a3c24cc0'::uuid)))
                            ->  Bitmap Index Scan on ipam_prefix_prefix_length_84660485  (cost=0.00..183.29 rows=10001 width=0)
                                  Index Cond: (prefix_length < ipam_prefix.prefix_length)
        SubPlan 2
          ->  Limit  (cost=784.33..784.34 rows=1 width=8)
                ->  Aggregate  (cost=784.33..784.34 rows=1 width=8)
                      ->  Bitmap Heap Scan on ipam_prefix u0_1  (cost=183.30..784.32 rows=6 width=0)
                            Recheck Cond: (prefix_length > ipam_prefix.prefix_length)
                            Filter: ((network >= ipam_prefix.network) AND (broadcast <= ipam_prefix.broadcast) AND (COALESCE(vrf_id, '3def19a1-48cb-4fa9-a98a-5e48a3c24cc0'::uuid) = COALESCE(ipam_prefix.vrf_id, '3def19a1-48cb-4fa9-a98a-5e48a3c24cc0'::uuid)))
                            ->  Bitmap Index Scan on ipam_prefix_prefix_length_84660485  (cost=0.00..183.29 rows=10001 width=0)
                                  Index Cond: (prefix_length > ipam_prefix.prefix_length)

Because there are no direct relationships between prefixes, a full sequential scan is needed, followed by further index scans for parents and children. I don't see a way around that in the current data model.

The key to the slowness is the limit in this case. Meaning, try the difference between these two and you will see where the problem is (provided you have this much data):

Prefix.objects.annotate_tree()
Prefix.objects.annotate_tree()[29700:29750]