Add experimental end-to-end test framework

[rob05c]

What does this PR do?

Fixes #(issue_number)

Which TC components are affected by this PR?

• Documentation
• Grove
• Traffic Analytics
• Traffic Monitor
• Traffic Ops
• Traffic Ops ORT
• Traffic Portal
• Traffic Router
• Traffic Stats
• Traffic Vault
• Other _________

What is the best way to verify this PR?

Check all that apply

• This PR includes tests
• This PR includes documentation updates
• This PR includes an update to CHANGELOG.md
• This PR includes all required license headers
• This PR includes a database migration (ensure that migration sequence is correct)
• This PR fixes a serious security flaw. Read more: www.apache.org/security

[asfgit]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/2834/
Test PASSed.

[asfgit]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/2836/
Test PASSed.

[asfgit]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/2843/
Test FAILed.

[asfgit]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/2844/
Test PASSed.

[asfgit]

Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/trafficcontrol-PR/2845/
Test PASSed.

[limited]

Have you looked at using the same autest framework that Traffic Server uses?
https://bitbucket.org/autestsuite/reusable-gold-testing-system

[rob05c]

I didn't look at autest in particular, but I did look at a number of others. This does use go test, which gives us a limited framework and does some of the basic work for us. I'm not opposed to auttest, but I'm not sure what the benefits would be? It looks like it's more oriented to the command line, as opposed to code. Which again, I don't have a strong opinion either way, if there's an advantage.

I did look at several HTTP-level frameworks (Selenium, Katalon, SoapUI). I was actually really hoping we could use one of them, to reduce writing framework-level stuff. But I came to the conclusion that there are enough CDN-specific things we'd want to test, that any framework at that level would either be incapable of, or we'd end up using its scripting language to essentially write our own framework within in.

When I say "CDN-specific", I mean things like verifying a DNS DS got a direct IP to the cache from a host lookup, or that a video could play. A lot of it is actually more network-specific, if we could find a good "network e2e test framework" that might work; but I didn't see one. Likewise for video.

[limited]

I was thinking that autest already had a solid trafficserver integration written, but I'm reconsidering because a Traffic Control E2E test would talk primarily to Traffic Ops. We also get that pretty much for free in this framework by using the TO client library.

I've got some more thoughts on what types of functionality an E2E framework should have based on some experiences with our own in house regression tests. I'll try to write those up a bit later.

In general, I'm favorable on improving TC test infrastructure, I think it could make our lives a lot easier and increase quality

[rob05c]

a Traffic Control E2E test would talk primarily to Traffic Ops

Well, it needs to to get the data, to make requests. But after that, its primary job should be making requests thru the Router/Edge. If we felt strongly about a non-Go test, I think it'd be ok to write a thin client in that language to just auth and the endpoints we need.

I've got some more thoughts on what types of functionality an E2E framework should have based on some experiences with our own in house regression tests. I'll try to write those up a bit later.

+1 We can use all the input we can get for this kind of thing. No one person is going to think of all the cases we should test.

based on some experiences with our own in house regression tests.

This is a total tangent, but while you're here, I have some other ideas about a "Data Validation" tool I think we should write. TC has a ton of data constraints that the CDN simply doesn't function without. I've said since the beginning, I think those data constraints should be encoded in the DB; but there's some pretty strong resistance to that. So, I had an idea about the next best thing: a tool to validate them. (This is obviously similar to Voluspa). For example, Cache profiles MUST have a health.polling.url parameter. There are countless data requirements like that. I think a tool to just get data from TO and verify requirements like that would be really easy to write, and be a big safety/confidence gain. It could be run all the time - every build, test upgrade, people could run it every 5 minutes in production even.

[limited]

Yeah I agree there is a lack of validation in the CDN config. There are too many things that could and often do go wrong.

I definitely think we could use improvement within Traffic Ops as far as imposing/checking constraints. My favorite tool for constraint satisfaction is the database- I haven't heard the reasoning behind the resistance to that yet.

Regardless, this is a place our users constantly shoot themselves in the foot (or we shoot them in the foot with migration errors) and can certainly be improved. That tool you are proposing is probably a good first step and hopefully can be pretty simple

[limited]

@rob05c Here are a few really rough notes, didn't have time to write full sentences and explanations around them

Components of an end to end test bed

Goals

• Well documented and extensible - must be very easy for newcomers to 1) run existing tests and 2) add a test.
• Speed - There may eventually be hundreds/thousands of tests in here. Three days for a full run is too long- most tests should run on the order of seconds including setup and teardown.

Setup

• Minimize external dependencies
• Separate test execution from resources
  • Flexibility is still important, should be able to run against existing testbed (bare metal) or create VMs if needed without impacting the test’s topologies and services (really useful for CI or getting new users started with the framework)
• For a CDN a complete framework might be able to create and manipulate servers, create caches, DS, even origin servers and content
• Failure injection is critical, even into external systems like DNS and the network.

Test Definition
Prerequisities - Can individual test require certain CDN topology and content be present. Can framework create resources if not present (pytest does this pretty well with its idea of fixtures)

Execution - Lots of common actions between, i.e making HTTP or DNS requests. Should be very simple to make a request but extensible if needed. i.e. default headers are necessary, but easy to add new headers. Similarly it should be really easy to make sure an HTTP request succeeded. If needed, more detailed validation should be possible if the test-writer needs.