Custom Backend/Route update performance

[galen0624]

@murphymj25 and I have been working to improve the route update performance in Fabio. We have a very large set of services (+4,000), routes (+10,000) and TPS (+120,000 across 8 fabios) . Due to the very large set of services/route updates its taking 30-60 seconds to perform the route updates. Even with the concurrent serviceMonitors that we added a while back (#564) set at a high level (100+) we still have the slow updates.

To improve the performance we have added a custom backend that is a simple API call to a remote system. The return values map directly to the route.RouteDef struct. This eliminates the text parsing and a number of other functions only needed with the consul backend. This could be used by other Fabio users who don't deploy Consul but have the ability to stand up an API for quick updates.

API return json schema

  [
    {
      "cmd": "string",
     "service": "string",
      "src": "string",
       "dest": "string",
     "weight": float,
    "tags": ["string"],
       "opts": {"string":"string"}
    }
   ]

We have built a caching service that takes the consul data and populates the API. Performance is significantly better but we still run into an issue with the updates taking around 15 seconds. Most of that time seems to be with

fabio/route/route.go

Line 52 in 3de70c7

if t.Service == service && t.URL.String() == targetURL.String() && t.FixedWeight == fixedWeight && reflect.DeepEqual(t.Tags, tags) {

I suspect the DeepEqual is the culprit and I wanted to understand if the dedup is required if we trust our inbound data sources? Or is this used to solve other issues? If we remove the dedup call the update for 10,000 routes goes from 15 seconds to 0.2 seconds.

We are nearing completion of our testing and will be submitting a PR. We wanted to start getting your feedback on the issue and subsequent custom backend addition.

[galen0624]

Code for your review - https://github.com/fabiolb/fabio/tree/issue-611-custom-backend

[galen0624]

make test file for your review
issue-611-make-test.txt

[galen0624]

After further testing the dedup isn't an issue as we had our 50K routes installed incorrectly. The dedub was crawling those as if it was a single service. Once we have the proper data being ingested the table load times are subsecond for 50K routes. Just for fun the table load took 2 seconds for 170K routes and 20 seconds for 1.7M routes. 😄

[galen0624]

@magiconair @leprechau Any thoughts on this idea/PR? Any improvements/changes we can make?
We most likely will move forward internally with this approach but would like to stay as close to the public master branch as possible.

[aaronhurt]

Sorry, I'll try to get a review in on this today or tomorrow.

[galen0624]

No worries. I know everyone is crazy busy.

[aaronhurt]

@galen0624 Sorry for the delay. Didn't get to looking this over last week.

This is a great feature and I can see it being a great way to expand the capabilities of Fabio without having to continuing to modify the core for other route sources.

One small niggle ... on the struct name can we make CustomBE just Custom so it matches the style of the existing Static, File, and Consul in that package?

Other than that LGTM and thank you again for your contributions and patience!

[galen0624]

@leprechau Sounds good. I'll update next week. Out of town at the moment. Thanks for taking a look. I know this isn't your day job so I appreciate the work you put in.

[galen0624]

#614 Updated CustomBE to Custom