Skip to content
This repository has been archived by the owner on Nov 13, 2018. It is now read-only.

Traffic_Ops_20

Jump to bottom
Jan van Doorn edited this page Sep 11, 2016 · 53 revisions

The below links may be stale. Use the [Traffic Control 3](Traffic Control 3) pages for new features and plans after the Postgres migration

What we are trying to do

The current MySQL<->Perl/Mojo<-HTML->jQuery-in-browser has a number of drawbacks:

  1. Perl/mojo is (really) hard to maintain
  2. There is a lot of duplicated work maintaining a UI and an API on the server side
  3. The single database support of MySQL is not really open
  4. Code conventions (table names, col names, etc) have slipped, and it's not clear anymore what the "proper" way to do things is.

Our proposed solution: Change to PostgreSQL<->golang<-JSON->Angular.JS

Feature Parity

Not all features in TO 1.0 are being used... Some can be removed from TO because there are off the shelf tools that do the function better, some can be removed because they never really worked, and we found other ways to get the job done, and some were a bad idea from the beginning. Going through them menu-item by menu-item:

Feature 2.0 Notes
Health->Table View
Health->Graph View Move all graphs and dashboards to Grafana, and just link from TO
Health->Server Checks Current status is broken, and the backing database schema is very ugly. Need to discuss more.
Health->Daily Summary Move all graphs and dashboards to Grafana, and just link from TO
Deliveryservices Some details are going to change, but we need CRUD for delivery services
DS->Federation
Servers
Servers->Upload CSV
Parameters->* Parameter table is getting an overhaul.
Generate ISO Need to discuss more. Is this really used? Aren't there smarter ways to achieve this?
Queue Updates Config state machine will make this obsolete?
DB Dump Very handy, but is it needed in the TO UI? (I think so)
Snapshot CRConfig Config state machine will make this obsolete?
Invalidate Content
Manage DNSSEC Keys
Misc->* With the below exceptions
Misc->Hardware Too Dell specific, and too messy. Not part of core TO functionality
Misc->ASNs Need a way to do this that other people can understand
Misc->Data Types Reviewing type table usage
Change Log
Help->* Really start using the release notes tab?

Thoughts on PostgreSQL and the conversion from MySQL

So first off, yes... proposal is to move everything from MySQL to PostgreSQL and to not support both databases, but have a migration, and have 1.x use MySQL and 2.x and beyond use PostgreSQL. How is that better just supporting MySQL? The MySQL license is incompatible with ours and PostgreSQL's is not. See https://www.wikivs.com/wiki/MySQL_vs_PostgreSQL for more info.

We will create a "migration" script that takes the data from MySQL in 1.x and enters it into a 2.x PostgreSQL database with the 2.0 (cleaned up) schema, so the migration strategy is a hard cut-over.

High availability

TO 2.0 SHALL support high availability as a core feature, not as an afterthought or bolt-on. Implementation details TBD.

Thoughts on golang and "frameworks"

Since we are building an API on the server side and not serving up generated HTML, we can stick to the Go stdlib net/http package with simple http.Handler's called from http.ListenAndServeTLS(). The most "frameworky" would be importing something like gorilla-mux but Go really shies away from frameworks in general.

The high level direction is to have the database do more of the work, and create a thin application/middleware layer in golang. For instance, the complicated queries that are needed to create the CRConfig.json (or equivalent info set for Traffic Router) that are now in Perl/DBIx code will be defined as views in the database, and the golang code just does a select out of those views. The structs needed to deal with these views in golang and convert them to json are verbose, but easily generated using the database as the source of truth.

Current thinking is that we do a one-time gen of code so we have a starting point, but from there on out we edit the genned files as if they're real code, and we don't go back and regen. We may use the generator for adding new tables / views, but they'll be cut-and-paste, not generate and replace. This one time gen will obviously only generate the table CRUD methods, but that's better than nothing.

Thoughts about the API

(started by jmitchell and dew)

  1. we need support for the GET, POST, PUT, DELETE http verbs where GET = list, POST = create, PUT = update, DELETE = delete

  2. An api route should communicate 2 things: the "path" should identify the RESOURCE and the "http verb" should identify the ACTION being taken (on the resource)

examples:

  • GET /api/v2/delivery_services <-- action=list, resource=all delivery services
  • GET /api/v2/delivery_services/:id <-- action=list, resource = delivery services where id = :id

  1. we want to stick to snakecase in the API routes

    • GOOD: /api/v2/delivery_services
    • BAD: /api/v2/deliveryServices

  2. we want to use the plural form for resources in the API:

    • GOOD: /api/v2/delivery_services
    • GOOD: /api/v2/delivery_services/:id
    • BAD: /api/v2/delivery_service
    • BAD: /api/v2/delivery_service/:id

  3. API versions should be dictated in the URL (unless latest version is desired). Only the major number will be indicated, since only major revision changes will not be backward compatible. Examples:

  • GET /api/v2/delivery_services
  • GET /api/delivery_services/:id <-- uses the latest version of the API
  1. we want to limit "route parameters" to id only
  • GOOD: GET /api/v2/delivery_services/:id
  • BAD: GET /api/v2/delivery_services/type/:type

  1. Any "filters" (besides id) should be represented as query parameters instead of route parameters.

    • GOOD: GET /api/v2/delivery_services?type=4
    • BAD: GET /api/v2/delivery_services/type/:type

    Think of query parameters as the parts of the where clause (if applicable).

    • select * from deliveryservice where type = 4

    Only a limited set of 'filters' will be supported, to be evaluated on a per view basis e.g. servers table - list by location or cache group but not by interface_name, every filter should have a well thought out use-case.

  2. Pagination and sorting will not be supported initially when retrieving collections - will revisit when this becomes a problem or in v3.

  3. In most cases GET, POST, PUT, DELETE should suffice:

  • GOOD: PUT /api/v2/delivery_services/:id
  • BAD: POST /api/v2/delivery_services/:id/update

  1. However, there may be exceptions where you want to perform an action that is not a list, create, update or delete in which this is acceptable. Use cases will be evaluated on a case by case basis.

  2. json should be the default, so no need to specify it

  • GOOD: GET /api/v2/deliveryServices/:id
  • Not necessary but won't hurt anything: GET /api/v2/deliveryServices/:id.json
  1. Specifying fields would be a nice to have but will not be implemented for v2. Will be revisited as needed, or deferred to v3.
  • GET /api/v1.2/delivery_services/:id?fields=xmlId,displayName
  1. Response objects should look like http://tools.ietf.org/html/draft-kelly-json-hal-07, specifically for linked things (type, status, profile, etc, and self) Example:
{
"response": [
 {
   "id": 1,
   "asn": 13367,
   "lastUpdated": "2012-09-17T15:41:22Z",
   "_links": {
     "_self": "/api/2.0/asn/1",
     "cachegroup": {
       "id": 29,
       "_ref": "/api/2.0/cachegroup/29"
     }
   }
 },
 { .. },
..
],
"version": 2,
"alerts": [
 {
   "level": "success",
   "text": "108 rows returned."
 }
]
}

  1. Booleans should be presented as true|false instead of 0|1, and numbers should be numbers, not strings

  2. Use swagger for API documentation, testing and validation

  3. When updating the entire object should be posted

  4. How do we namespace "public" vs. "internal" apis?

Thoughts on Roles / Permissions

Rethink how roles are implemented. Consider a role a grouping of api endpoints. Ex: Admin role can use all api endpoints, Ops role can use a subset of api endpoints...etc, etc. See how functions are grouped by roles here as an example: https://codex.wordpress.org/Roles_and_Capabilities

#####Roles proposal:

######6 roles

  • Super Admin - Can do everything across all delivery services

  • Ops Admin - Can do almost everything across all delivery services

  • Ops Read-only - Can view everything across all delivery services

  • Tenant Admin - Can do everything within the scope of their assigned delivery services

  • Tenant Ops - Can do everything an Admin can do, but can't CRUD Sub-Tenants or Users

  • Tenant Read-Only - Can view everything within the scope of their assigned delivery services, can create reports and notifications

Note: If you login as LDAP, you'll be assigned the Ops Read-Only role

Note: No "disallowed" role. Instead user.active can be true|false

######Requirements:

  • A tenant table
  • A tenant_id FK in the deliveryservices table
  • A User <--> Tenants join table to assign Tenants to users (only necessary for Tenant* roles)
  • Multiple roles for a user is possible. Even having a different role for each tenant is possible. example:

Cindy | Tenant Admin | Tenant1

Cindy | Tenant Read-Only | Tenant2

######Nice to have:

Hierarchy of tenants. example:

  • Parent Tenant
    • Child Tenant1
    • Child Tenant2

Cindy | Tenant Admin | Parent Tenant

^^ this means Cindy is a Tenant Admin of Parent Tenant, Child Tenant1 and Child Tenant2

Thoughts on the client (TO UI)

The current Perl/Mojo<-HTML->jQuery-in-browser implementation of Traffic Ops tightly couples a significant portion of data processing with the presentation layer (HTML). With TO 2.0, the presentation layer and the data layer will be split into two distinct applications.

  • Application #1 - TO API 2.0 built in Golang serves as a request-response message system expressed in JSON. It is responsible for fetching / processing data through publicly exposed endpoints. Presentation is no longer a concern of the server.

  • Application #2 - TO UI 2.0 built in HTML, CSS, Javascript is responsible for handling the presentation and interaction with the data provided from the TO API. The TO UI is nothing more than one of potentially many API consumers.

By seperating concerns between the 2 applications (UI presentation / markup (TO UI) and fetching / processing data (TO API)), parallel development efforts become achievable.

Optionally, a SPA (single page application) framework may be employed (i.e. AngularJS) to provide a better user experience. SPA applications load all the HTML, CSS and Javascript resources at once and don't rely on subsequent page loads or refreshes.

Conventions

Database
  1. Everything in the database is lowercase and snake_case
  2. Tables are singular so servers not server
  3. Links (foreign keys) to other tables are with that table name. so server has a cachegroup column that links to cachegroup.id, not a cachegroup_id column. Note: this is pending the surrogate key decision
  4. All tables have a surrogate primary key called id (including the linking tables like profile_parameter) Under review - Phil to create a "natural key" schema to see how that feels
  5. Need to decide on value of created_at and updated_at column
API

See API section

Code

For the go server: all in with "golint and go vet" (even though some of those rules seem pedantic).

TODO / Don't forget:

Database / schema cleanup

See details page

  • Decide on customer app / system operator app split. Same App, different privs, or different app?
    • From that the user/role definitions will make sense

Open issues / Questions

  • Come up with standard for ingesting and exporting data
    • Example: Import server csv app/lib/ui/UploadServerCsv.pm

See our website for official docs.

[Traffic Control 3](Traffic Control 3)

[Traffic Control 2.0](Traffic Control 2.0)

Projects

Design discussions

Presentations

[Notes and Processes](Notes and Processes)

Clone this wiki locally