Add Cache-Side Config Generator (#3762)

* Add client-side ATS config gen

Adds a client-side ATS config gen interceptor to ORT, with an initial
parent.config gen and passing everything else to TO. Plan is to
add all configs to be client-side generated.

* Add TO deliveryserviceserver ids query params

Also adds a client function to specify server and DS IDs, as well as
a missing client func for the existing deliveryservice?cdn param.

* Add ORT atstccfg to ort build scripts

* Add ORT args to override hostname, OS check

For debugging, or emergencies.

* Add ORT using atstccfg instead of TO directly

Also changes atstccfg to do things ORT needed:

 - changed to return the HTTP code as the exit code, on error
 - added CLI option to return which configs are generated (vs proxied)
 - added retry num option, for failed TO attempts
 - changed to use lib/tc-log, and take args for where to log
 - fixed missing license headers
 - added integrity check via SHA512 or Content-length headers

* Add atstccfg insecure, timeout, cache age flags

* Fix merge conflict

* Add TO ORT gen profile configs

* Add atstccfg changelog

* Add atstccfg docs

* Add license for vendored pflag

* Remove old/wrong comments

* Remove unused symbols

* Removed docs comma

* Fix license files

.dependency_license

@@ -101,6 +101,7 @@ jquery\.tree\.min\.css$, MIT
 jquery\.dataTables\..*\.(css|js)$, MIT
 github\.com/basho/backoff/.*, MIT
 github\.com/dchest/siphash/.*, CC0
+github\.com/pkg/errors\..*, BSD
 traffic_portal/app/src/assets/js/chartjs/angular-chart\..*, BSD
 traffic_portal/app/src/assets/css/jsonformatter\..*, Apache
 traffic_portal/app/src/assets/js/jsonformatter\..*, Apache

CHANGELOG.md

@@ -41,6 +41,7 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).
 - Added pagination support to some Traffic Ops endpoints via three new query parameters, limit and offset/page
 - Traffic Ops now supports a "sortOrder" query parameter on some endpoints to return API responses in descending order
 - Traffic Ops now uses a consistent format for audit logs across all Go endpoints
+- Added cache-side config generator, atstccfg, installed with ORT. Currently includes parent.config and all profile configs, proxies others to Traffic Ops.
 
 ### Changed
 - Traffic Router:  TR will now allow steering DSs and steering target DSs to have RGB enabled. (fixes #3910)

LICENSE

@@ -440,3 +440,11 @@ For the lestrrat-go/jwx (commit e35178a) component:
 For the dgrijalva/jwt-go (commit 5e25c22) component:
 @traffic_ops/traffic_ops_golang/vendor/github.com/dgrijalva/jwt-go/*
 ./traffic_ops/traffic_ops_golang/vendor/github.com/dgrijalva/jwt-go/LICENSE
+
+For the ogier/pflag (commit 45c278a) component:
+@vendor/github.com/ogier/pflag/*
+./licenses/BSD-pflag
+
+For the errors (commit 27936f6) component:
+@vendor/github.com/pkg/errors/*
+./vendor/github.com/pkg/errors/LICENSE

docs/source/admin/traffic_server.rst

@@ -80,6 +80,21 @@ Configuring Traffic Server
 ==========================
 All of the :abbr:`ATS (Apache Traffic Server)` application configuration files are generated by Traffic Ops and installed by :term:`ORT`. The :file:`traffic_ops_ort.pl` file should be installed on all :term:`cache server` s (See `Installing the ORT Script`_), usually in ``/opt/ort``. It is used to do the initial install of the configuration files when the :term:`cache server` is being deployed, and to keep the configuration files up-to-date when the :term:`cache server` is already in service.
 
+.. _config-generation:
+
+ORT Config File Generation
+-------------------------
+
+In the past, ATS config files were generated by Traffic Ops. Traffic Control is in the process of moving ATS config file generation to a library for generic use, and to an application which uses the library and resides on the cache.
+
+The library, ``lib/go-atscfg``, allows users to write their own applications and servers, if they wish to generate ATS configuration files and deploy them to caches via other means. For example, if you wish to generate config files with an additional service, or continue generating config files on Traffic Ops itself via a plugin or local service.
+
+The app, ``atstccfg``, is installed by the traffic_ops_ort RPM alongside the ORT script. This app makes standard API calls to Traffic Ops, and uses their data to build the ATS config files. The ORT script now requests all config through the app, which generates config files it has, and requests directly from Traffic Ops the files it doesn't recognize.
+
+This provides several benefits. Primarily, reduces the overhead and risk of the monolithic Traffic Ops installation and upgrade process, and allows operators to canary-test config changes one cache at a time, and in the event of an error, only rolling back a few canary caches rather than the entire Traffic Ops instance.
+
+In order to see which config files are generated by a given ``ORT`` or ``atstccfg`` version, run ``/opt/ort/atstccfg --print-generated-files``.
+
 .. _installing-ort:
 
 Installing the ORT Script

lib/go-atscfg/astatsdotconfig.go

@@ -0,0 +1,38 @@
+package atscfg
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+const AstatsSeparator = "="
+const AstatsFileName = "astats.config"
+
+func MakeAStatsDotConfig(
+	profileName string,
+	paramData map[string]string, // GetProfileParamData(tx, profile.ID, AstatsFileName)
+	toToolName string, // tm.toolname global parameter (TODO: cache itself?)
+	toURL string, // tm.url global parameter (TODO: cache itself?)
+) string {
+	hdr := GenericHeaderComment(profileName, toToolName, toURL)
+	txt := GenericProfileConfig(paramData, AstatsSeparator)
+	if txt == "" {
+		txt = "\n" // If no params exist, don't send "not found," but an empty file. We know the profile exists.
+	}
+	txt = hdr + txt
+	return txt
+}

lib/go-atscfg/atscfg.go

@@ -0,0 +1,111 @@
+package atscfg
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+	"errors"
+	"strconv"
+	"strings"
+	"time"
+
+	"github.com/apache/trafficcontrol/lib/go-tc"
+)
+
+const HeaderCommentDateFormat = "Mon Jan 2 15:04:05 MST 2006"
+
+type ServerInfo struct {
+	CacheGroupID                  int
+	CDN                           tc.CDNName
+	CDNID                         int
+	DomainName                    string
+	HostName                      string
+	ID                            int
+	IP                            string
+	ParentCacheGroupID            int
+	ParentCacheGroupType          string
+	ProfileID                     ProfileID
+	ProfileName                   string
+	Port                          int
+	SecondaryParentCacheGroupID   int
+	SecondaryParentCacheGroupType string
+	Type                          string
+}
+
+func (s *ServerInfo) IsTopLevelCache() bool {
+	return (s.ParentCacheGroupType == tc.CacheGroupOriginTypeName || s.ParentCacheGroupID == InvalidID) &&
+		(s.SecondaryParentCacheGroupType == tc.CacheGroupOriginTypeName || s.SecondaryParentCacheGroupID == InvalidID)
+}
+
+func HeaderCommentWithTOVersionStr(name string, nameVersionStr string) string {
+	return "# DO NOT EDIT - Generated for " + name + " by " + nameVersionStr + " on " + time.Now().Format(HeaderCommentDateFormat) + "\n"
+}
+
+func GetNameVersionStringFromToolNameAndURL(toolName string, url string) string {
+	return toolName + " (" + url + ")"
+}
+
+func GenericHeaderComment(name string, toolName string, url string) string {
+	return HeaderCommentWithTOVersionStr(name, GetNameVersionStringFromToolNameAndURL(toolName, url))
+}
+
+// GetATSMajorVersionFromATSVersion returns the major version of the given profile's package trafficserver parameter.
+// The atsVersion is typically a Parameter on the Server's Profile, with the configFile "package" name "trafficserver".
+// Returns an error if atsVersion is empty or not a number.
+func GetATSMajorVersionFromATSVersion(atsVersion string) (int, error) {
+	if len(atsVersion) == 0 {
+		return 0, errors.New("ats version missing")
+	}
+	atsMajorVer, err := strconv.Atoi(atsVersion[:1])
+	if err != nil {
+		return 0, errors.New("ats version parameter '" + atsVersion + "' is not a number")
+	}
+	return atsMajorVer, nil
+}
+
+type DeliveryServiceID int
+type ProfileID int
+type ServerID int
+
+// GenericProfileConfig generates a generic profile config text, from the profile's parameters with the given config file name.
+// This does not include a header comment, because a generic config may not use a number sign as a comment.
+// If you need a header comment, it can be added manually via ats.HeaderComment, or automatically with WithProfileDataHdr.
+func GenericProfileConfig(
+	paramData map[string]string, // GetProfileParamData(tx, profileID, fileName)
+	separator string,
+) string {
+	text := ""
+	for name, val := range paramData {
+		name = trimParamUnderscoreNumSuffix(name)
+		text += name + separator + val + "\n"
+	}
+	return text
+}
+
+// trimParamUnderscoreNumSuffix removes any trailing "__[0-9]+" and returns the trimmed string.
+func trimParamUnderscoreNumSuffix(paramName string) string {
+	underscorePos := strings.LastIndex(paramName, `__`)
+	if underscorePos == -1 {
+		return paramName
+	}
+	if _, err := strconv.ParseFloat(paramName[underscorePos+2:], 64); err != nil {
+		return paramName
+	}
+	return paramName[:underscorePos]
+}

lib/go-atscfg/atscfg_test.go

@@ -0,0 +1,52 @@
+package atscfg
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestGenericHeaderComment(t *testing.T) {
+	objName := "foo"
+	toolName := "bar"
+	toURL := "url"
+
+	txt := GenericHeaderComment(objName, toolName, toURL)
+
+	testComment(t, txt, objName, toolName, toURL)
+}
+
+func testComment(t *testing.T, txt string, objName string, toolName string, toURL string) {
+	commentLine := strings.SplitN(txt, "\n", 2)[0] // SplitN always returns at least 1 element, no need to check len before indexing
+
+	if !strings.HasPrefix(strings.TrimSpace(commentLine), "#") {
+		t.Errorf("expected comment on first line, actual: '" + commentLine + "'")
+	}
+	if !strings.Contains(commentLine, toURL) {
+		t.Errorf("expected toolName '" + toolName + "' in comment, actual: '" + commentLine + "'")
+	}
+	if !strings.Contains(commentLine, toURL) {
+		t.Errorf("expected toURL '" + toURL + "' in comment, actual: '" + commentLine + "'")
+	}
+	if !strings.Contains(commentLine, objName) {
+		t.Errorf("expected profile '" + objName + "' in comment, actual: '" + commentLine + "'")
+	}
+}

lib/go-atscfg/atsdotrules.go

@@ -0,0 +1,53 @@
+package atscfg
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+	"strings"
+)
+
+func MakeATSDotRules(
+	profileName string,
+	paramData map[string]string, // GetProfileParamData(tx, profile.ID, StorageFileName)
+	toToolName string, // tm.toolname global parameter (TODO: cache itself?)
+	toURL string, // tm.url global parameter (TODO: cache itself?)
+) string {
+	text := GenericHeaderComment(profileName, toToolName, toURL)
+
+	drivePrefix := strings.TrimPrefix(paramData["Drive_Prefix"], `/dev/`)
+	drivePostfix := strings.Split(paramData["Drive_Letters"], ",")
+
+	for _, l := range drivePostfix {
+		l = strings.TrimSpace(l)
+		if l == "" {
+			continue
+		}
+		text += `KERNEL=="` + drivePrefix + l + `", OWNER="ats"` + "\n"
+	}
+	if ramPrefix, ok := paramData["RAM_Drive_Prefix"]; ok {
+		ramPrefix = strings.TrimPrefix(ramPrefix, `/dev/`)
+		ramPostfix := strings.Split(paramData["RAM_Drive_Letters"], ",")
+		for _, l := range ramPostfix {
+			text += `KERNEL=="` + ramPrefix + l + `", OWNER="ats"` + "\n"
+		}
+	}
+
+	return text
+}

lib/go-atscfg/atsdotrules_test.go

@@ -0,0 +1,52 @@
+package atscfg
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+import (
+	"strings"
+	"testing"
+)
+
+func TestMakeATSDotRules(t *testing.T) {
+	profileName := "myProfile"
+	toolName := "myToolName"
+	toURL := "https://myto.example.net"
+	paramData := map[string]string{
+		"Drive_Prefix":      "/dev/sd",
+		"Drive_Letters":     "a,b,c,d,e",
+		"RAM_Drive_Prefix":  "/dev/ra",
+		"RAM_Drive_Letters": "f,g,h",
+	}
+
+	txt := MakeATSDotRules(profileName, paramData, toolName, toURL)
+
+	testComment(t, txt, profileName, toolName, toURL)
+
+	if count := strings.Count(txt, "\n"); count != 9 { // one line for each drive letter, plus 1 comment
+		t.Errorf("expected one line for each drive letter plus a comment, actual: '%v' count %v", txt, count)
+	}
+
+	if !strings.Contains(txt, "sda") {
+		t.Errorf("expected sda for drive letter, actual: '%v'", txt)
+	}
+	if !strings.Contains(txt, "rah") {
+		t.Errorf("expected sda for drive letter, actual: '%v'", txt)
+	}
+}