A comprehensive suite of BackstopJS scenarios to automate visual regression testing on CiviCRM

⚠️ Please note that currently the suite works only on Drupal-based CiviCRM sites. Wordpress and/or Joomla sites are not supported yet

Requirements

A CiviCRM site with sample data (check the "Load sample data" option when running the installer)

Technical

• Node.js v14.16.0
• Drush installed globally

Extensions (Not Required when using Github Action)

• The uk.co.vedaconsulting.mosaico extension should not be enabled on the site. In case it is, either momentarily disable it or remove the mailings_menu.json scenarios group

Drupal

• Clean URLs enabled
• The user name for admin should be admin

How to use it

Using Github Actions

1. Go to Actions Tab
2. Click on 'Run Workflow' button.
3. Enter required 'Reference CiviCRM Version', 'Test CiviCRM Version', 'Shoreditch Reference Branch/Tag Name' and 'Shoreditch Test Branch/Tag Name'.
4. Wait for the Action to finish, and download the Artifacts to compare the screenshots.

Manual Process

1. Create a backstop_data/site-config.json file with the following content.

   {
     "url": "your_local_url",
     "root": "absolute_path_to_site"
   }

2. Setup required data

   npx gulp backstopjs:setup-data

3. Create the reference screenshots

   npx gulp backstopjs:reference

4. Create the test screenshots and compare

   npx gulp backstopjs:test

Testing a single group

In case you want to run backstop only for the scenarios of a specific group (ie. administer-menu.json), pass the --group argument to the gulp task (without the .json extension)

npx gulp backstopjs:test --group administer-menu

Parallel capturing

BackstopJS supports taking multiple screenshot at once. Change the value of asyncCaptureLimit in backstop.tpl.json to decide how many screenshots you want to take in parallel

Note:Please be aware that BackstopJS performance is heavily dependent on the specs of the machine it runs on, so make sure to choose a value that the tool can handle on your machine (otherwise you will encounter random timeout errors)

Cookies

This backstop test suite relies on session cookies to be present in the backstop_data/cookies/admin.jsonfile. The file is automatically generated by the gulp task by using the drush uli command.

If you want to skip the cookie creation phase (in case the cookies are already present in the folder and are not expired yet), pass a --skipCookies argument to the task

npx gulp backstopjs:reference --skipCookies

Covered pages

Search menu

• Find Contacts
• Advanced search
• Standard search results
• Advanced search results
• Full-text Search
• Full-text Search results
• Search Builder
• Find Contributions
• Find Mailings
• Find Membership
• Find Participants
• Find Participants results
• Find Pledges
• Find Activities
• Custom Searches

Contacts menu

• New Individual
• New Household
• New Organization
• Contact Reports
• New Activity
• New Email
• Import Contacts
• Import Activities
• New Group
• Manage Groups
• New Tag
• Manage Tags (Categories)
• Find and Merge Duplicate Contacts

Contributions menu

• Dashboard
• New Contribution
• Find Contribution
• Contribution Reports
• Import Contributions
• Pledges - Dashboard
• Pledges - New Pledges
• Pledges - Find Pledges
• Pledges - Pledges Reports
• Pledges - New Pledge Report
• Batch Data Entry
• New Batch Data Entry
• Accouting Batch - New Batch
• Accouting Batch - Open Batches
• Accouting Batch - Closed Batches
• Accouting Batch - Exported Batches
• New Contribution Page
• Manage Contribution Pages
• Personal Campaign Pages
• Premiums (Thank-you Gifts)
• New Price Set
• Manage Price Set
• Close Accounting Period

Events menu

• Dashboard
• Register Event Participant
• Find Participants
• Event Reports
• Import Participants
• New Event
• Manage Events

Mailings menu

• New Mailings
• Draft and Unscheduled Mailings
• Scheduled and Sent Mailings
• Archived Mailings
• Mailing Reports
• Headers, Footers, and Automated Messages
• Headers, Footers, and Automated Messages - Add Mailing Component
• Message Templates
• From Email Addresses
• New SMS
• Find Mass SMS
• New A/B Test
• Manage A/B Tests

Memberships menu

• Dashboard
• New Membership
• Find Memberships
• Membership Reports
• Batch Data Entry
• Import Memberships

Administer menu

Administration Console

• System Status
• Configuration Checklist

Customize Data and Screens

• Custom Fields
• New Custom Field
• Profiles
  • Name and Address - CiviCRM Profile Fields
• Tags
• Activity Types
• Relationship Types
• Contact Types
• Dropdown Options
  • Gender Options
  • Individual contact prefixes Options
  • Individual contact suffixes Options
  • Instant Messenger (IM) screen-names Options
  • Location Types (Home, Work...)
  • Mobile Phone Providers Options
  • Phone Type Options
  • Website Type Options
• Display Preferences
• Search Preferences
• Date Preferences
• Navigation Preferences
• World Replacements
• Manage Custom Searches

Communications

• Organization Address and Contact Info
• Message Templates
• New Message Template
• Schedule Reminders
• Preferred Communication Methods
• Label Formats
• Print Page (PDF) Formats
• Communication Style Options
• Email Greeting Type Options
• Postal Greeting Type Options
• Addressee Type Options

Localization

• Settings
• Addresses
• Date
• Languages Options

Users and Permissions

• Access Control
• Synchronize Users to Contacts

System Settings

• Enable Components
• Connections
• Extensions
• Cleanup Caches and Update Paths
• Debugging and Error Handling
• Upload Directories
• Import/Export Mappings
• Mapping and Geocoding Providers
• Misc
• Option Groups
• Payment Processor
• Resource URLs
• Safe File Extension Options
• Scheduled Jobs
• SMS Provider

CiviReport

• All Reports
• Create New Report from Template
• Manage Templates
• Register Report

CiviMember

• Membership Types
• Membership Status Rules
• Component Settings

CiviCampaign

• Survey Types
• Campaign Types
• Campaign Status
• Engagement Index
• CiviCampaign Component Settings

CiviContribute

• Financial Types
• Financial Accounts
• Payment Methods Options
• Accepted Credit Cards Options
• Soft Credit Types Options
• Payment Processor
• Payment Processor

CiviEvent

• Event Templates
• Event Type Options
• Participant Status
• Participant Role Options
• Participant Listing Options
• Event Name Badge Layouts
• New Event Name Badge Layout
• CiviEvent Component Settings

CiviMail

• Mail Accounts
• CiviMail Settings
• CiviMail Component Settings