Add Cache-Side Config Generator #3762

rob05c · 2019-07-29T21:01:50Z

What does this PR (Pull Request) do?

Adds a cache-side config generator. This is a small go binary (atstccfg), which ORT calls instead of requesting Traffic Ops. The atstccfg generator then generates configs it can, and proxies to TO for configs it can't yet generate.

This PR includes generating parent.config and all profile configs, with everything else being proxied. The plan is to move configs over one at a time, until all configs are generated cache-side.

See https://cwiki.apache.org/confluence/display/TC/Cache-Side+Config+Generation

It also moves the logic into /lib/go-atscfg, a library that can be used if someone wants to generate configs themselves somewhere else.

I've parity-tested the generated configs, against Traffic Ops generation, on all our production edges, mids, and profiles. They all diff identical (except where order is nondeterministic in Perl).

Includes documentation, explaining the new config generation, what's generated where and how.
Includes unit tests.
Includes Changelog.

This PR is not related to any other Issue

Which Traffic Control components are affected by this PR?

Traffic Ops
Traffic Ops ORT

What is the best way to verify this PR?

Install ORT, run, verify config files match previous versions / TO.

If this is a bug fix, what versions of Traffic Control are affected?

Not a bug fix.

The following criteria are ALL met by this PR

This PR includes tests OR I have explained why tests are unnecessary
This PR includes documentation OR I have explained why documentation is unnecessary
This PR includes an update to CHANGELOG.md OR such an update is not necessary
This PR includes any and all required license headers
This PR ensures that database migration sequence is correct OR this PR does not include a database migration
This PR DOES NOT FIX A SERIOUS SECURITY VULNERABILITY (see the Apache Software Foundation's security guidelines for details)

asfgit · 2019-07-29T22:11:09Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/4052/
Test FAILed.

asfgit · 2019-08-06T17:30:41Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/4107/
Test FAILed.

asfgit · 2019-08-07T21:13:04Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/4108/
Test FAILed.

asfgit · 2019-08-09T20:01:02Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/4110/
Test FAILed.

asfgit · 2019-08-09T22:50:14Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/4111/
Test FAILed.

asfgit · 2019-08-09T23:00:52Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/4112/
Test FAILed.

docs/source/admin/traffic_server.rst

lib/go-atscfg/volumedotconfig.go

ezelkow1

Looks good, ran some comparison tests and go tests. Docs could just use a rewrite

rob05c · 2019-08-29T16:51:21Z

Addressed PR comments, fixed merge conflicts.

asf-ci · 2019-08-29T17:28:29Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/4190/
Test FAILed.

rob05c · 2019-08-29T17:32:03Z

retest this please

asf-ci · 2019-08-29T17:51:39Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/4192/
Test FAILed.

asf-ci · 2019-09-03T23:30:31Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/4195/
Test FAILed.

ocket8888 · 2019-09-06T16:46:35Z

License header problems:

Error                                 Unknown! traffic_ops/ort/atstccfg/urisigningconfig_test.go
Error                                 Unknown! traffic_ops/ort/atstccfg/urlsigconfig_test.go

ocket8888 · 2019-09-06T17:13:50Z

It also doesn't like the license of your pkg dependency:

Error                                    BSD~! vendor/github.com/pkg/errors/.travis.yml
Error                                     BSD! vendor/github.com/pkg/errors/LICENSE
Error                                    BSD~! vendor/github.com/pkg/errors/Makefile
Error                                    BSD~! vendor/github.com/pkg/errors/appveyor.yml
Error                                    BSD~! vendor/github.com/pkg/errors/bench_test.go
Error                                    BSD~! vendor/github.com/pkg/errors/errors.go
Error                                    BSD~! vendor/github.com/pkg/errors/errors_test.go
Error                                    BSD~! vendor/github.com/pkg/errors/example_test.go
Error                                    BSD~! vendor/github.com/pkg/errors/format_test.go
Error                                    BSD~! vendor/github.com/pkg/errors/json_test.go
Error                                    BSD~! vendor/github.com/pkg/errors/stack.go
Error                                    BSD~! vendor/github.com/pkg/errors/stack_test.go

LICENSE

rawlinp · 2019-09-06T17:19:57Z

Where did we land on the idea to share code between the new atstccfg tool and the API and just replacing DB queries w/ API calls? Are we doing that?

asf-ci · 2019-09-17T18:50:31Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/4264/
Test PASSed.

rob05c · 2019-09-17T20:42:45Z

License header problems

Fixed.

rob05c · 2019-09-17T20:44:46Z

Where did we land on the idea to share code between the new atstccfg tool and the API and just replacing DB queries w/ API calls? Are we doing that?

All the logic is shared, in /lib/go-atscfg. The data loading is still using regular DB calls, though. I didn't see a clean way to use the API within TO. It's technically possible, but it would be much slower, and load lots of unnecessary data.

asf-ci · 2019-09-17T21:04:09Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/4269/
Test FAILed.

asf-ci · 2019-09-17T21:08:17Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/4270/
Test PASSed.

rawlinp · 2019-09-17T21:21:10Z

All the logic is shared, in /lib/go-atscfg. The data loading is still using regular DB calls, though. I didn't see a clean way to use the API within TO. It's technically possible, but it would be much slower, and load lots of unnecessary data.

Yeah, I don't think it's necessary to try to use the API within TO. Thanks!

asf-ci · 2019-09-18T00:03:15Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/4273/
Test PASSed.

asf-ci · 2019-09-18T22:45:27Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/4278/
Test PASSed.

Adds a client-side ATS config gen interceptor to ORT, with an initial parent.config gen and passing everything else to TO. Plan is to add all configs to be client-side generated.

Also adds a client function to specify server and DS IDs, as well as a missing client func for the existing deliveryservice?cdn param.

For debugging, or emergencies.

Also changes atstccfg to do things ORT needed: - changed to return the HTTP code as the exit code, on error - added CLI option to return which configs are generated (vs proxied) - added retry num option, for failed TO attempts - changed to use lib/tc-log, and take args for where to log - fixed missing license headers - added integrity check via SHA512 or Content-length headers

asf-ci · 2019-09-20T05:04:02Z

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/4294/
Test PASSed.

rob05c added Traffic Ops related to Traffic Ops cache-config Cache config generation labels Jul 29, 2019

rob05c force-pushed the to-cache-side-configs branch from ee858cf to c4151ab Compare August 6, 2019 15:53

rob05c force-pushed the to-cache-side-configs branch from c4151ab to 1a85575 Compare August 7, 2019 20:40

rob05c force-pushed the to-cache-side-configs branch from 1a85575 to 181f2fa Compare August 9, 2019 19:55

rob05c added the new feature A new feature, capability or behavior label Aug 9, 2019

rob05c marked this pull request as ready for review August 9, 2019 20:03

ezelkow1 reviewed Aug 13, 2019

View reviewed changes

docs/source/admin/traffic_server.rst Outdated Show resolved Hide resolved

ezelkow1 reviewed Aug 27, 2019

View reviewed changes

lib/go-atscfg/volumedotconfig.go Show resolved Hide resolved

ezelkow1 suggested changes Aug 27, 2019

View reviewed changes

rob05c force-pushed the to-cache-side-configs branch from 78d721f to 0092849 Compare August 29, 2019 16:50

rob05c mentioned this pull request Aug 29, 2019

Add TO Go ATS CDN configs #3875

Merged

7 tasks

rob05c force-pushed the to-cache-side-configs branch from 0092849 to 0a35305 Compare September 3, 2019 19:28

ezelkow1 approved these changes Sep 6, 2019

View reviewed changes

rawlinp reviewed Sep 6, 2019

View reviewed changes

LICENSE Outdated Show resolved Hide resolved

rob05c force-pushed the to-cache-side-configs branch 2 times, most recently from a0dcbe7 to e22d392 Compare September 17, 2019 20:36

rob05c force-pushed the to-cache-side-configs branch from e22d392 to d6c1ca9 Compare September 17, 2019 22:29

rob05c force-pushed the to-cache-side-configs branch from d6c1ca9 to d1497df Compare September 18, 2019 22:37

rob05c mentioned this pull request Sep 18, 2019

Add TO Go remap.config #3688

Merged

7 tasks

rob05c added 15 commits September 19, 2019 21:48

Add client-side ATS config gen

7efd2a9

Adds a client-side ATS config gen interceptor to ORT, with an initial parent.config gen and passing everything else to TO. Plan is to add all configs to be client-side generated.

Add TO deliveryserviceserver ids query params

5cfb32d

Also adds a client function to specify server and DS IDs, as well as a missing client func for the existing deliveryservice?cdn param.

Add ORT atstccfg to ort build scripts

2cf51c9

Add ORT args to override hostname, OS check

7075537

For debugging, or emergencies.

Add atstccfg insecure, timeout, cache age flags

fce1e27

Fix merge conflict

8ae52ce

Add TO ORT gen profile configs

73d1e4b

Add atstccfg changelog

9ed58f4

Add atstccfg docs

a3bbc72

Add license for vendored pflag

e5bb4b7

Remove old/wrong comments

6dbed4d

Remove unused symbols

bca0759

Removed docs comma

896b4f1

Fix license files

d5bd5f3

rob05c force-pushed the to-cache-side-configs branch from d1497df to d5bd5f3 Compare September 20, 2019 03:48

mitchell852 merged commit dc9bdf5 into apache:master Sep 25, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Cache-Side Config Generator #3762

Add Cache-Side Config Generator #3762

rob05c commented Jul 29, 2019 •

edited

asfgit commented Jul 29, 2019

asfgit commented Aug 6, 2019

asfgit commented Aug 7, 2019

asfgit commented Aug 9, 2019

asfgit commented Aug 9, 2019

asfgit commented Aug 9, 2019

ezelkow1 left a comment

rob05c commented Aug 29, 2019

asf-ci commented Aug 29, 2019

rob05c commented Aug 29, 2019

asf-ci commented Aug 29, 2019

asf-ci commented Sep 3, 2019

ocket8888 commented Sep 6, 2019

ocket8888 commented Sep 6, 2019

rawlinp commented Sep 6, 2019

asf-ci commented Sep 17, 2019

rob05c commented Sep 17, 2019 •

edited

rob05c commented Sep 17, 2019

asf-ci commented Sep 17, 2019

asf-ci commented Sep 17, 2019

rawlinp commented Sep 17, 2019

asf-ci commented Sep 18, 2019

asf-ci commented Sep 18, 2019

asf-ci commented Sep 20, 2019

Add Cache-Side Config Generator #3762

Add Cache-Side Config Generator #3762

Conversation

rob05c commented Jul 29, 2019 • edited

What does this PR (Pull Request) do?

Which Traffic Control components are affected by this PR?

What is the best way to verify this PR?

If this is a bug fix, what versions of Traffic Control are affected?

The following criteria are ALL met by this PR

asfgit commented Jul 29, 2019

asfgit commented Aug 6, 2019

asfgit commented Aug 7, 2019

asfgit commented Aug 9, 2019

asfgit commented Aug 9, 2019

asfgit commented Aug 9, 2019

ezelkow1 left a comment

Choose a reason for hiding this comment

rob05c commented Aug 29, 2019

asf-ci commented Aug 29, 2019

rob05c commented Aug 29, 2019

asf-ci commented Aug 29, 2019

asf-ci commented Sep 3, 2019

ocket8888 commented Sep 6, 2019

ocket8888 commented Sep 6, 2019

rawlinp commented Sep 6, 2019

asf-ci commented Sep 17, 2019

rob05c commented Sep 17, 2019 • edited

rob05c commented Sep 17, 2019

asf-ci commented Sep 17, 2019

asf-ci commented Sep 17, 2019

rawlinp commented Sep 17, 2019

asf-ci commented Sep 18, 2019

asf-ci commented Sep 18, 2019

asf-ci commented Sep 20, 2019

rob05c commented Jul 29, 2019 •

edited

rob05c commented Sep 17, 2019 •

edited