bug 1431259: add headless tests for cdn #4765
Conversation
There was a problem hiding this comment.
Thanks @escattone, this looks like a good framework for testing these headers. It will be next week before I can dig in.
I think this would be a good opportunity to use a dynamic fixture. If the _kuma_status.json call returns headers like x-cache that show the webserver is behind CloudFront, then expect CloudFront caching headers. Otherwise, expect s-maxage, etc.
tests/headless/test_cdn.py
Outdated
| return response | ||
|
|
||
|
|
||
| @pytest.mark.headless |
There was a problem hiding this comment.
Do you use these mark tags? I haven't incorporated them into my workflow yet.
There was a problem hiding this comment.
No, I don't use them. I was like a sheep following along with the other headless tests! 🐑 In fact, now that I think of it, I think we should dump them, since I don't think they're really that useful. What do you think?
There was a problem hiding this comment.
Let's remove them until we use them. I thought nondestructive was used somewhere (Jenkins?), but I think it may be the same a login tests and unused.
|
@jwhitlock That's a good idea to use a dynamic fixture. I'll look into that. |
|
I force pushed an update. |
Codecov Report
@@ Coverage Diff @@
## master #4765 +/- ##
=======================================
Coverage 95.82% 95.82%
=======================================
Files 270 270
Lines 24568 24568
Branches 1750 1750
=======================================
Hits 23542 23542
Misses 814 814
Partials 212 212Continue to review full report at Codecov.
|
There was a problem hiding this comment.
An update to pytest (3.1.0 seems to work) is needed, or remove pytest.param.
I ran pytest tests/headless --reruns=1 --base-url=<url to test>. Retries were needed to deal with staging 504s. This ran with no errors with --base-url=https://developer.mozilla.org and --base-url=https://developer.allizom.org.
With --base-url=http://localhost:8000, I got errors for:
- /diagrams/workflow/workflow.svg
- /presentations/microsummaries/index.html
- /en-US/docs/Web/HTML$revision/1293895
- /en-US/docs/Web/HTML$compare?locale=en-US&to=1299417&from=1293895
- /en-US/docs/Learn/CSS/Styling_text/Fundamentals#Color
- /en-US/docs/Learn/CSS/Styling_text/Fundamentals$toc
- /files/2767/hut.jpg
- /@api/deki/files/3613/=hut.jpg
I think some can be fixed by ensuring those pages are in the sample database, but others (legacy files, attachments) won't work in the development environment by default. I think that follow-on work can be done in a new PR, and I can do it.
| @pytest.mark.nondestructive | ||
| @pytest.mark.parametrize( | ||
| 'slug, params', | ||
| [pytest.param('/en-US/dashboards/topic_lookup', {'topic': 'mathml'}, |
There was a problem hiding this comment.
pytest.param is not in pytest 3.0.7. I think it was introduced in pytest 3.1.0.
There was a problem hiding this comment.
Oops, thanks! I was using pytest (3.5.1) locally!
| @pytest.mark.parametrize( | ||
| 'slug, params', | ||
| [pytest.param('/en-US/dashboards/topic_lookup', {'topic': 'mathml'}, | ||
| marks=pytest.mark.skip), # Gets 504 due to slow response |
There was a problem hiding this comment.
What's your reasoning for adding a test that we are skipping from the start?
There was a problem hiding this comment.
I wanted to make this an xfail, since most of the time the endpoint is so slow the CDN returns a 504 (the CDN is already configured at the maximum timeout of 60 seconds), but that adds over a minute to the tests, so I thought I'd add it as a skip as a reminder that this endpoint has issues. Perhaps it's best to remove for now, and put this information in a # TODO: ... comment?
There was a problem hiding this comment.
I could go with skip or comment it out. Skip makes it more likely we'll remember to unskip it later, if we improve performance of topic_lookup. A comment means you don't have to update pytest.
There was a problem hiding this comment.
I'll go the route of updating pytest, thanks @jwhitlock.
|
This is ready for review again @jwhitlock. Sorry, I force-pushed a rebase by habit, but had wanted to add a separate commit for the |
| pytest==3.0.7 \ | ||
| --hash=sha256:66f332ae62593b874a648b10a8cb106bfdacd2c6288ed7dec3713c3a808a6017 \ | ||
| --hash=sha256:b70696ebd1a5e6b627e7e3ac1365a4bc60aaf3495e843c1e70448966c5224cab | ||
| # Code: https://github.com/pytest-dev/pytest |
This is the state of the browserless CDN tests after about 3 days. The tests cover every behavior in the CDN except for
static/*, and cover a significant portion of the endpoints. There are certainly more tests that could be developed to check specific endpoints for cache variation based on specific headers or cookies or query parameters, but these are a decent start I think.During the course of writing these tests, I found another bug with the current configuration of the CDN, namely that when editing based on a revision (e.g., https://developer.allizom.org/en-US/docs/Web/HTML$edit/1299417) the publish will be met with a permission-denied error. This is due to the fact that this endpoint bypasses the
*$editbehavior, a pass-through behavior which forwards thecsrftokencookie, and falls instead into the default behavior, which does not forward thecsrftokencookie. I filed this bug as bugzilla #1459087.