Implement M3 server

[rjb1000]

Feature description

Using OpenAPI YAML interface definition files, implement an M3 server for the Python-based Application Server

• Provision and manage the life-cycle of Server Certificate resources.
  • Create, Retrieve, Update, Destroy.
• Provision and manage the life-cycle of Content Hosting Configuration resources.
  • Create, Retrieve, Update, Destroy.
  • Purge the associated content cache.

Cache purge

For the purposes of the reference implementation using Nginx (which only supports purging individual cache objects, all cache objects under a URL prefix, or all cache objects) only the following very limited subset of regular expressions is supported:

• Most regular expression reserved characters not allowed.
  • Only terminal .* allowed, '.' or '*' is not allowed anywhere else in the regular expression unless escaped with \.
  • Regular expression not allowed to start with ^ (start of string anchor).
  • \ may be used to escape characters that would normally be interpreted as regular expression operators.
• Regular expression must be a relative path, not an absolute path.
  • Regular expression is evaluated relative to the AF-defined base URL path.

In case the regular expression does not meet the constrained specification above, a suitable error message (e.g. 400 Bad Request) should be returned at M3 along with a human-readable error message, e.g. "Regular expression not supported by this implementation". (This error will, in future, be propagated by the 5GMS AF to the M1 client.)

Relevant specifications and corresponding sections

TS 26.512 plus pre-standardisation M3 Open API YAML interface definition files.

Additional context

This will allow the 5GMSd AS to be managed by a 5GMSd AF (see 5G-MAG/rt-5gms-application-function#7) or by a test client (see #31). As a consequence, the ability to read a Content Hosting Configuration from a local file should be removed from the 5GMSd AS as part of the scope.

It will be possible for multiple Content Hosting Configurations to be simultaneously provisioned in the 5GMSd AS.

[davidjwbbc]

Added M3 interface to the application server on my fork (feature-add-m3-interface branch), and am just running through some tests using curl to mock up client requests and view the responses.

Looking good so far.

[davidjwbbc]

I've run through various tests (adding/removing/updating certificates/content-hosting-configurations) and the M3 interface appears to work. There's a bug with passing the certificate filename to the nginx configuration that needs fixing. The only operation I have left to look at is the cache purge operation, which at present will return a Not Implemented error.

From what I remember from previously reading about nginx and purging cache entries is that this is done by handling PURGE requests and redirecting to a local nginx service which will expire the cache objects. As I remember it this operates either on specific cache entries or on the whole cache at once. I need to do some reading up on current nginx configuration to see how fine grained these cache purges can be made to be as I suspect we won't be able to provide the full level of control specified in TS 26.512.

[rjb1000]

It would be worthwhile investigating whether Apache, Varnish, etc. offer the flexibility to purge entries based on a regular expression as well as Nginx, @davidjwbbc.

If this requirement of TS 26.512 cannot easily be met by existing caching web proxies, we should inform 3GPP SA4 and ask them to think again about this feature.

[davidjwbbc]

During testing of 5G-MAG/rt-5gms-application-function#7, it was noticed that the application server currently only handles HTTP/1.1 for its M3 service. Need to look into how the Python server can support HTTP/2 with prior knowledge on this interface.

[davidjwbbc]

From reading up on the uvicorn github issues about HTTP/2 support, it looks like simply switching from using the uvicorn module to the hypercorn module will enable support for HTTP/1.1 and HTTP/2 (and optionally HTTP/3 if the hypercorn[h3] module installation option is used). I'll modify my current solution to change to the hypercorn python module as the ASGI server.

[davidjwbbc]

So using hypercorn, the M3 interface now supports HTTP/1.1, HTTP/2 upgrade and HTTP/2 prior knowledge.

[davidjwbbc]

I've been looking at cache purging facilities available with different web server/proxy softwares. This is in order to understand what's possible with each one with respect to implementing the cache purging strategy defined in the 5G specifications.

Here is a table of 4 projects and their cache purging capabilities:

Software	Purge Object	Purge Whole Cache	Purge Cache Subtree	Purge Cache By Regex
NGINX	⛔1	⛔1	⛔1	⛔1
NGINX Plus	👍	👍	👍	⛔
Varnish	👍	👍2	⛔	⛔
Apache HTTPd	👍3	👍3	👍3	👍3
Squid	👍	👍2	⛔	⛔

All pieces of software are able to purge individual items from their cache.

Two of the web proxies can purge the whole cache while running, two require the daemon to be stopped and the cache deleted.

NGINX can purge a subtree of cache items by using a wildcard symbol (*) at the end of a cache key prefix, all items with the same prefix will be purged. To do this in apache the htcacheclean command can be used to list all cache entries, filter those that start with a given prefix and use the htcacheclean command to delete each one individually. There is a note on the Squid FAQs suggesting they would like to implement purge by wildcard but that this is not currently available.

Only Apache HTTPd can be made to purge by regular expression by using the htcacheclean command to list all entries, filter the list using the regular expression and then use htcacheclean to delete each item which matched the filter.

Footnotes

1. Not available as part of the software but can be emulated by using an external script to filter and delete cache files and then "reloading" the daemon. ↩ ↩² ↩³ ↩⁴

2. Achieved by stopping the proxy daemon, deleting the cache and restarting the daemon. ↩ ↩²

3. Using the external htcacheclean command to list all cache entries and/or purge individual cached objects. ↩ ↩² ↩³ ↩⁴

[rjb1000]

Thanks for the detailed research above, @davidjwbbc.

For the purposes of our reference implementation of the cache purge operation using Nginx, I suggest that we support a very limited subset of regular expressions. I have amended the issue description to specify a minimal subset corresponding to Nginx capabilities. (Implementations using different web proxies may implement a different subset of regular expressions, of course.)

[davidjwbbc]

I've had an issue implementing the cache purge for NGINX.

So the part I missed when researching the table above is the fact that the cache purger in NGINX is only available in the commercial version (even though the documentation is included in the free version), the free version doesn't have a cache purging option built in.

A little looking around the internet suggests other people simply grep for the URL they wish to purge in all files in the cache directory and delete any matching files, then signal NGINX to reload its files. It's not a very nice way of doing it and relies on NGINX storing the cache key in plain text in the cache object files. But this is something we could do in Python. If we're going down this route then this also gives us the freedom to perform the operation using full regular expressions.

[davidjwbbc]

To implement manually we can look for [LF]KEY: <cache-key>[LF] near the start of the cache file (string appears 336 bytes into the file in the NGINX cache objects I've looked at). Reading the first 4KB of the file and looking for the cache key should do it. If we make sure the cache key contains the provisioning session id followed by the request URL path to the file being requested, then we can ensure that only entries for the correct session are removed. This can be done by using the proxy_cache_key directive in the nginx configuration.

[davidjwbbc]

Implemented the purge operation in the Python wrapper script as detailed above. The operation ends in a HUP signal to NGINX if any cache files have been deleted.

Tested this using the test m3_client_cli.py command in both purge all and purge regex modes of operation. Cache purged successfully.

I will let @devbbc do some testing against the rt-5gms-application-function with M3 interface. If no problems found then I'll tidy up the commits and generate a PR.

[devbbc]

Tested the working of the M3 interface by running this Application Server against the M3 client enabled rt-5gms-application-function. No problems found in this test and they worked as expected.

[davidjwbbc]

While looking at TS 26.512 for 5G-MAG/rt-5gms-application-function#15, I noticed that the cache should also be purged when a Content Hosting Configuration is destroyed at reference point M1 on the 5GMS AF (or if the parent Provisioning Session is destroyed resulting in the Content Hosting Configuration subresource being destroyed).

I've therefore implemented this so that the NGINX cache is purged of all items in the path of the Content Hosting Configuration when it is destroyed by the 5GMS AF via reference point M3.