Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proxy route API discussion #796

Closed
dormando opened this issue Jun 10, 2021 · 5 comments
Closed

Proxy route API discussion #796

dormando opened this issue Jun 10, 2021 · 5 comments
Labels
idea work goal

Comments

@dormando
Copy link
Member

dormando commented Jun 10, 2021

This issue is for a discussion related to #716 - the embedded cache proxy.

Most of the proxy is C, two parts are handled in lua:

  1. Loading the configuration of servers and pools (referred to as hash selectors), as well as configuring the route layout.
  2. Executing routes, which take a request object and matches it with a pool, optionally checking the response. The C side then hashes, executes, reads the response, then returns to lua.

An example configuration file exists here: https://github.com/memcached/memcached/pull/716/files?file-filters%5B%5D=.lua&file-filters%5B%5D=No+extension#diff-1ab37d6d2cf900c8b5df02081cba299ebf0356af8a54f6111cedae0f527a81b0 - I'll explain the basics here as well.

To load the configuration, a dedicated thread first compiles the lua code. Then calls the function mcp_config_pools, which loads all backends, configures pools, and returns a table holding hash selectors. Once completed it will, for each worker thread, execute mcp_config_routes. This function is expected to set up route handling (code that matches requests to a pool), and sets the command hooks that memcached will call (ie; hooks on get, set, and so on).

The proxy flow starts by parsing a request (ie: get foo) and looking for a function hook for this command. If a hook exists, it will call the supplied function. If no hook exists, it will handle the request as though it were a normal memcached.

In lua, this looks like: mcp.attach(mcp.CMD_GET, function) - Functions are objects and can be passed as arguments. The function is called within a coroutine, which allows us to designs routes procedurally even if they have to make a network call in the middle of executing.

The function is called with a prototype of:

function(request)

end

The most basic example of a valid route would be:
mcp.attach(mcp.CMD_GET, function(r) return "SERVER_ERROR no route\r\n" end)

For any get command, we will return the above string to the client. This isn't very useful as-is. We want to test the key and send the command to a specific backend pool; but the function only takes a request object. How are routes actually built?

The way we recommend configuring routes are with function closures. In lua functions can be created capturing the environment they were created in. For example:

function new_route()
  local res = "SERVER_ERROR no route\r\n"
  return function(r)
    return res
  end
end

mcp.attach(mcp.CMD_GET, new_route())

In this example, new_route() returns a function. This function has access to the environment (local res = ) of its parent. When proxy calls the CMD_GET hook, it's calling the function that was returned by new_route(), not new_route() itself. This function uselessly returns a string.

This should give you enough context to understand the example routes listed in the startfile.lua link above. To break it down a little more:

  • pools (hash selectors) are special functions, when called in the same function(r) end format, they match a request to a specific backend server, pause the lua function, executes the command, then resumes the original lua function.
  • In the example file servers are "hard coded", but they do not have to be! Lua can load other lua files, parse data files, or you can make calls to a network service.
  • Since routes and pools use the same function prototype, you can route to a route just as easily as you can route to a pool. This lets us compose routes, ie; "check the prefix of the key, then send to the next route, which checks for a command type, then sends to another route that checks a backup server on a miss, which finally sends to a pool".

Since we have a real programming language for both configuration and the routes themselves, we can write loops around patterns and keep the configuration short.


My goal is to work through some main concepts:

  1. requests and response objects are custom lua objects. I need to figure out the API for these objects (ie; r:key() returns the request key to lua, and r:hit() says if a response was a HIT or not). We should be able to modify or replace the key, make new requests, and examine a response in detail.
  2. A standard library of route handlers. The startfile.lua has some high level examples that I wrote quickly, but this needs to be fleshed out for common use cases, as well as simply for code examples. People with complex or unique setups are encouraged to create their own route handlers. There is nothing requiring you to use the standard library!
  3. It comes up a lot so I'll note here: Observability (logs, stats, etc) are in planning.

Also of note: Custom hash selectors are possible, if you don't want to use one of the default ones. See proxy/ketama for an example.

If you're using mcrouter or some similar memcached proxy, please take a look and let us know what's important to you!

@dormando
Copy link
Member Author

dormando commented Sep 29, 2021

I've used this configuration system to create... a simple configuration wrapper, which is what most end users will expect to use: https://github.com/dormando/memcached/blob/proxy_preview/proxy/simple.lua

example configs:

-- by defaults, sends "/foo/*" to "foo" and "/bar/*" to "bar"
pool{
        name = "foo",
        backends = {"127.0.0.1:11212", "127.0.0.1:11213"},
}

pool{
        name = "bar",
        backends = {"127.0.0.1:11214", "127.0.0.1:11215"},
}

with multiple "zones" per pool:

-- if my_zone() is commented out, will look for "backends" instead of "zones"
my_zone("z1")

-- NOTE: optional to specify this if defaults are okay.
router{
    router_type = "keyprefix",
        match_prefix = "/(%a+)/",
        default_pool = nil
}

-- NOTE: a normal config will have backends _or_ zones, not both!
pool{
        name = "foo",
        backends = {"127.0.0.1:11212", "127.0.0.1:11213"},
        zones = {
      z1 = {
                "127.0.0.1:11212",
        "127.0.0.1:11213",
      },
      z2 = {
                "127.0.0.1:11214",
        "127.0.0.1:11215",
          },
    }
}

This is still lua syntax, but easily could be YAML. I'll be primarily working on this "simple" interface myself. The actual route functions the "simple" config uses might be ported to a higher level library, that people can use directly when the simple configuration doesn't work for them.

IE: it should be possible to do tiering, collect specific stats counters, etc via customizing this library or writing your own. If "simple" doesn't do it for you, go nuts.

@dormando dormando mentioned this issue Sep 29, 2021
19 tasks
@TysonAndre
Copy link
Contributor

If you're using mcrouter or some similar memcached proxy, please take a look and let us know what's important to you!

One thing I use frequently is a local cache in front of memcached when I'm locally developing against a staging version of an application that's physically far away (100ms).

Currently, I already solved this for the application itself - this is done manually in the application in an in-memory cache, but I can see that being useful elsewhere.

  • e.g. work as a read-through cache, potentially with a shorter lifetime, if inconsistency is permittible
  • e.g. convert sets to remote invalidation attempts if the application created them from potentially stale data of the local embedded memcache proxy cache (if the local version and the remote version differed)

@goldfishy
Copy link

This proxy looks very interesting! Two questions:

  1. We’ve (engineering team at Spotify) been running some load-tests on mcrouter and were able to push it to roughly 10-15 k RPS / core[1] with one network hop in between proxy and shards for the most rudimentary cluster topologies (single proxy with a single downstream replica). Do you have any estimates / ideas roughly how this proxy will perform?
  2. Got any estimates when you think the proxy might reach production maturity?

[1] n2d machines on GCP

@dormando
Copy link
Member Author

Hey @goldfishy - thanks for expressing interest!

  1. Performance should be considerably better than that per core, but it will depend on what you're doing and what your latency tolerances are. Right now the only thread topology uses a shared IO thread for backend handling; if I push things in a benchmark that seems to top out somewhere between 250k-500k rps. This should crawl higher with tuning, and I'll also be adding a mode which uses one backend connection per worker thread, which would offer unlimited scalability at the cost of more backend tcp connections.
  2. "Soon". I took a break from it for a few weeks since the other folks interested in it are all busy, so I wasn't getting much feedback to work on.

If you can spare any time the best thing towards getting it production ready faster is to give it a test; let me know what features are missing for you, what performance you get (and what the gap is between acceptable/top end), and any bugs/weirdness you run into while testing. Having someone actively testing will help a lot in motivating me to continue the production-ready bug scrub too :)

I did a lot of the pre-production cleanup work last month. It should be much much easier to build and try than mcrouter, just see the extra steps in BUILD.

Productionizing work is tracked in #827 - discussions about the lua API's should be done here though.

@dormando
Copy link
Member Author

Going to close out this issue - Documentation is being created at https://github.com/memcached/memcached/wiki/Proxy work still being tracked at #827 and the code is out in released tarballs (though users recommended to use the next branch if possible, as per wiki docs).

@goldfishy - if you folks are still interested, development picked up again this year and it should be pretty usable now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
idea work goal
Projects
None yet
Development

No branches or pull requests

3 participants