New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug 1462400: Improve functional tests #5143

Merged
merged 5 commits into from Nov 28, 2018

Conversation

Projects
None yet
2 participants
@escattone
Copy link
Member

escattone commented Nov 28, 2018

This PR is a step forward for the functional tests, but issues remain. For me, it felt the most natural to to bundle these five commits into one PR, but I can split them out into separate PR's if preferred.

Here are the highlights:

  • I've updated to these versions:
    • Python selenium package to version 3.141.0 (latest)
    • Python pytest-selenium package to version 1.14.0 (latest)
    • Docker image selenium/standalone-chrome:3.141.5 (latest)
    • Docker image selenium/standalone-firefox:3.141.5 (latest)
    • Python PyPOM package to version 2.2.0 (latest)
  • I moved away from using the selenium hub and selenium node Docker images to using the stand-alone selenium Docker images for chrome and firefox (resolved WebDriverException: Message: Error forwarding the new session cannot find : Capabilities {browserName: chrome, goog:chromeOptions: {args: [], extensions: []}} errors in the chrome tests)
  • I ran the stand-alone selenium images with --shm-size=2g to avoid browser crashes (this resolved some WebDriverException: Message: unknown error: session deleted because of page crash errors in chrome)
  • I moved to using docker-compose for the browser-based tests as a means of moving away from using the --link feature (which is discouraged) as well as for simplicity in managing the set-up and teardown.
  • I had to xfail (for now) the two manual macro search tests when run against Firefox (clicking either the "all" or "english-only" search buttons does nothing in Firefox, but works fine in Chrome). No other tests were removed from the test suite.
  • The test_footer_language_selector test remains problematic, as sometimes the payments popover obscures the ability to click. I was unable to fix that issue before my time ran out today.
  • Although some issues remain, almost all of the flakiness has been removed, so the tests run much more reliably.

@escattone escattone requested a review from jwhitlock Nov 28, 2018

@jwhitlock jwhitlock changed the title Improve functional tests 1462440 bug 1462400: Improve functional tests Nov 28, 2018

@escattone escattone referenced this pull request Nov 28, 2018

Closed

Fix integration/acceptance test suite #155

3 of 3 tasks complete
@jwhitlock
Copy link
Member

jwhitlock left a comment

The changes look reasonable, and a test run worked in Jenkins. There is still a flaky test due to the payments banner, but that can be worked on in a new PR. Thanks @escattone!

["pull": true,
"dockerfile": config.job.dockerfile])
}
try {

This comment has been minimized.

@jwhitlock

jwhitlock Nov 28, 2018

Member

I'm curious, how was this different from the previous version?

This comment has been minimized.

@escattone

escattone Nov 28, 2018

Member

Good question, and I'm sorry I didn't include my reasoning in the description of the PR. You're right, in effect it's not any different that what it replaces. My sole purpose was to start replacing our dependencies on https://github.com/mozmeao/jenkins-pipeline/ with code that's at least as simple but more direct, in that it doesn't require any special libraries. I think all of our uses of that library could be replaced with simple equivalents, and that we could eventually eliminate that dependency altogether.

// Setup the extra arguments for the pytest command.
def pytest_opts = ""
if (config.job && config.job.tests) {
pytest_opts += "-m '${config.job.tests}'"

This comment has been minimized.

@jwhitlock

jwhitlock Nov 28, 2018

Member

This is a little different from the previous version. The stage job include "not login", and these tests appear to have been skipped by the test collector

--capability browserName ${BROWSER:-chrome}
--base-url=${BASE_URL:-https://developer.allizom.org}
--junit-prefix=${BROWSER:-chrome}
--junit-xml=/app/test_results/functional-${BROWSER:-chrome}.xml

This comment has been minimized.

@jwhitlock

jwhitlock Nov 28, 2018

Member

I'm glad this made it. Jenkins shows the test counts, so it works.

# Provides facilities for defining, registering and looking up components.
# Code: https://github.com/zopefoundation/zope.component
# Changes: https://pypi.org/project/zope.component/#history
# Docs: https://zopecomponent.readthedocs.io/en/latest/

This comment has been minimized.

@jwhitlock

jwhitlock Nov 28, 2018

Member

Thanks for adding the metadata! 😍

@escattone escattone merged commit 42a2615 into mozilla:master Nov 28, 2018

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
security/snyk - package.json (mdn) No new, high severity issues
Details

@escattone escattone deleted the escattone:improve-functional-tests-1462440 branch Nov 28, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment