feature: new data model first steps (Routes & Services)

[thibaultcha]

Summary

This PR introduces the first steps a new DAO implementation and a new data model. Two new entities that are part of the new data model have been implemented first: Routes and Services. It contains work conducted by @bungle, @hishamhm, @kikito, and myself.

The new entities, Routes and Services, are introduced to decouple responsibilities previously held solely by the "API" entity. The API entity is still supported (making this PR backwards-compatible) but considered deprecated. The new DAO proposes a better implementation of database abstraction for both PostgreSQL and Cassandra, as well as a better interface for specifying entities schemas and nicer Admin API endpoints, addressing issues with the previous implementations of those components.

Table of Contents

• Routes and Services concepts
• Routes and Services schema
• Full changelog
• Issues resolved

Routes and Services concepts

By decoupling this entity, we hope to achieve a better separation of concerns in our model between what Kong considers as “downstream” and “upstream” in proxy jargon.

• Routes describes how Kong matches a request. Routes are the entry-point for clients (downstream), and contain routing rules (hosts, HTTP methods, paths, possibly with regexes).
• Services represents the user's upstream: these are the servers to which Kong proxies requests. A service has a host, port and possibly a path.

Or, with pictures, here is the previous model based on APIs:

And how Routes and Services decouples responsibilities:

The major benefit of Routes and Services is the ability to specify plugins on a subset of endpoints without duplicating the API definition (and the upstream_url property) and the shared plugin instances.

Note how the new model gets rid of redundant information: in the old model, both API objects had the same upstream_url attribute, and the key-auth plugin had to be applied to both entities, with redundant settings.

In the new model however, information pertaining each single route is stored in the Route entity, and upstream service information is stored in the Service entity. Note also that Plugin objects can be applied to either Routes (via route_id) or Services (via service_id). In the above example, the key-auth plugin attached to the Service will apply to all Routes that point to said Service.

Route and Services schema

Routes

Attribute	Description
id (primary key)	the route's UUID
created_at (auto-generated, read-only)	the creation timestamp
updated_at (auto-generated, read-only)	the update timestamp
protocols	a set of supported protocols, default is { "http", "https" }
methods (semi-optional - see below)	set of accepted HTTP methods (e.g. GET, POST, etc.)
hosts (semi-optional - see below)	an array of hostnames, supporting a wildcard at the beginning or end (examples: foo.com, foo.*, *.foo.com)
paths (semi-optional - see below)	an array of path components, which may use regexes (examples: /request, /foo[0-9]*)
regex_priority	an integer priority for sorting routing routes containing regular expressions
strip_path	boolean (same as in the old model)
preserve_host	boolean (same as in the old model)
service	a foreign key linking it to a service entity

At least one of methods, hosts or paths is required, in order to produce a routing rule for a Route.

Services

Attribute	Description
id (primary key)	the service's UUID
created_at (auto-generated, read-only)	the creation timestamp
updated_at (auto-generated, read-only)	the update timestamp
name (must be unique)	the service's name, giving it a friendlier identifier than the UUID
retries	integer (same as in the old model)
connect_timeout	integer (same as in the old model)
write_timeout	integer (same as in the old model)
read_timeout	integer (same as in the old model)
protocol (required)	the protocol used to connect to the service (default is “http”)
host (required)	the service hostname or IP
port	the service port (default is 80)
path	the service's path component
url	(write-only, convenience pseudo-field). If this field is set when creating/updating a service, its values are splitted and used to set the protocol, host, port and path fields.

Note: the fields protocol, host, port and path replace the field upstream_url in the old API entity, storing each fragment of the service URL separately. For convenience, the url field can be used to set those values at once.

Full changelog

• New Routes and Services entities with better separation of concerns
  • Plugins can be applied to Routes, Services, Consumers, or APIs (still) or a combination of all
  • Improved naming of our core entities and their attributes by using more generic/standard names
  • Routes with regex URIs have a priority field for modifying their evaluation order (APIs rely on their created_at field)
• New DAO layer
  • Reduce (not entirely) some harmful patterns such as read-before-writes, various un-optimized queries, etc...
  • Nicer interface for developers (although this the new DAO is not to be considered public/stable yet)
  • Better isolation between database-specific strategies: it will be easier for us to consider implementing support for new databases
  • Better extensibility for developers (although this the new DAO is not to be considered public/stable yet)
• New definitions for schema's entities
  • Enforcing purely declarative schemas, with more flexible rules that reduce the need for custom validation logic code
  • Schemas validate themselves via a "meta-schema": when this API will be public, we shall be able to catch developer errors when they write invalid schemas
• New Admin API endpoints, /routes and /services are their top-level prefixes. Those new endpoints have a nicer behavior:
  • Automatically generated CRUD endpoints for entities (with overriding capabilities)
  • Better response format (non-set fields are always visible via JSON NULL)
  • Better error messages: more consistent, more verbose, better suited for programmatic usage
  • Better support for compound types such as Arrays in both JSON and form-urlencoded content types (e.g. no more comma-separated arrays)
  • Ensure arrays in Lua do not get JSON-encoded as empty Objects {}, but empty Arrays []

Eventually, more entities will be moved to the new DAO layer and new Admin API/schemas. Those APIs will also be made available for public consumption by plugin developers as part of a later effort.

Issues resolved

TODO

[kswen]

so how does ring-balancer works with routes/services model?

there`s no targets node?

[thibaultcha]

@KwSen The load balancer hasn’t changed in behavior. It gets triggered for Services the same way it does for APIs as of 0.12: when the hostnames (of the API or Service in question) is the same as that of one of the registered Upstreams