Automated Visual Testing (2nd edition)

examples/visual_testing/ReadMe.md

@@ -1,7 +1,7 @@
 [<img src="https://cdn2.hubspot.net/hubfs/100006/images/SeleniumBaseText_F.png" title="SeleniumBase" align="center" height="38">](https://github.com/seleniumbase/SeleniumBase/blob/master/README.md)
 ### Automated Visual Testing (Layout Change Detection)
 
-Automated visual testing can help you detect when something has changed the layout of a web page. Rather than comparing screenshots, a more effective way is to detect layout differences by comparing HTML tags and properties. If a change is detected, it could mean that something broke on the webpage, or possibly something harmless like a website redesign or dynamic content.
+Automated visual testing helps you detect when the layout of a web page has changed. Rather than comparing screenshots, layout differences are detected by comparing HTML tags and properties with a baseline. If a change is detected, it could mean that something broke, the web page was redesigned, or dynamic content changed.
 
 To handle automated visual testing, SeleniumBase uses the ``self.check_window()`` method, which can set visual baselines for comparison and then compare the latest versions of web pages to the existing baseline.
 
@@ -16,7 +16,7 @@ After the first time ``self.check_window()`` is called, later calls will compare
 
 Here's an example call:
 ```
-self.check_window(name="first_test)", level=1)
+self.check_window(name="first_test)", level=3)
 ```
 On the first run (<i>or if the baseline is being set/reset</i>) the "level" doesn't matter because that's only used for comparing the current layout to the existing baseline.
 
@@ -26,12 +26,9 @@ Here's how the level system works:
 * level=1 ->
     HTML tags are compared to tags_level1.txt
 * level=2 ->
-    HTML tags are compared to tags_level1.txt and
-    HTML tags/attributes are compared to tags_level2.txt
+    HTML tags and attribute names are compared to tags_level2.txt
 * level=3 ->
-    HTML tags are compared to tags_level1.txt and
-    HTML tags + attributes are compared to tags_level2.txt and
-    HTML tags + attributes/values are compared to tags_level3.txt
+    HTML tags and attribute values are compared to tags_level3.txt
 
 As shown, Level-3 is the most strict, Level-1 is the least strict. If the comparisons from the latest window to the existing baseline don't match, the current test will fail, except for Level-0 tests.
 
@@ -44,7 +41,7 @@ As long as ``--visual_baseline`` is used on the command line while running tests
 
 If you want to use ``self.check_window()`` to compare a web page to a later version of itself from within the same test run, you can add the parameter ``baseline=True`` to the first time you call ``self.check_window()`` in a test to use that as the baseline. This only makes sense if you're calling ``self.check_window()`` more than once with the same "name" parameter in the same test.
 
-Automated Visual Testing with ``self.check_window()`` is not very effective for websites that have dynamic content because that changes the layout and structure of web pages. For those pages, you're much better off using regular SeleniumBase functional testing.
+Automated Visual Testing with ``self.check_window()`` is not very effective for websites that have dynamic content because that changes the layout and structure of web pages. For those pages, you're much better off using regular SeleniumBase functional testing, unless you can remove the dynamic content before performing the comparison, (such as by using ``self.ad_block()`` to remove dynamic ad content on a web page).
 
 Example usage of ``self.check_window()``:
 ```python
@@ -65,24 +62,77 @@ from seleniumbase import BaseCase
 
 class VisualLayoutTest(BaseCase):
 
-    def test_applitools_helloworld(self):
+    def test_applitools_layout_change(self):
         self.open('https://applitools.com/helloworld?diff1')
-        print('Creating baseline in "visual_baseline" folder...')
+        print('\nCreating baseline in "visual_baseline" folder...')
         self.check_window(name="helloworld", baseline=True)
         self.click('a[href="?diff1"]')
         # Verify html tags match previous version
         self.check_window(name="helloworld", level=1)
-        # Verify html tags + attributes match previous version
+        # Verify html tags and attribute names match previous version
         self.check_window(name="helloworld", level=2)
-        # Verify html tags + attributes + values match previous version
+        # Verify html tags and attribute values match previous version
         self.check_window(name="helloworld", level=3)
         # Change the page enough for a Level-3 comparison to fail
         self.click("button")
         self.check_window(name="helloworld", level=1)
         self.check_window(name="helloworld", level=2)
         with self.assertRaises(Exception):
             self.check_window(name="helloworld", level=3)
-        # Now that we know the exception was raised as expected,
+        # Now that we know the Exception was raised as expected,
         # let's print out the comparison results by running in Level-0.
+        # (NOTE: Running with level-0 will print but NOT raise an Exception.)
         self.check_window(name="helloworld", level=0)
 ```
+
+Here's the output of that:
+```
+AssertionError:
+First differing element 39:
+['div', [['class', ['section', 'hidden-section', 'image-section']]]]
+['div', [['class', ['section', 'image-section']]]]
+
+-  ['div', [['class', ['section', 'hidden-section', 'image-section']]]],
+?                                ------------------
++  ['div', [['class', ['section', 'image-section']]]],
+*
+*** Exception: <Level 3> Visual Diff Failure:
+* HTML tag attribute values don't match the baseline!
+```
+
+Here's another example:
+```python
+from seleniumbase import BaseCase
+
+
+class VisualLayoutTest(BaseCase):
+
+    def test_xkcd_layout_change(self):
+        self.open('https://xkcd.com/554/')
+        print('\nCreating baseline in "visual_baseline" folder.')
+        self.check_window(name="xkcd_554", baseline=True)
+        # Change height: (83 -> 130) , Change width: (185 -> 120)
+        self.set_attribute('[alt="xkcd.com logo"]', "height", "130")
+        self.set_attribute('[alt="xkcd.com logo"]', "width", "120")
+        self.check_window(name="xkcd_554", level=0)
+```
+
+Here's the output of that:
+```
+AssertionError:
+First differing element 22:
+['img[30 chars]['height', '83'], ['src', '/s/0b7742.png'], ['width', '185']]]
+['img[30 chars]['height', '130'], ['src', '/s/0b7742.png'], ['width', '120']]]
+
+-    ['height', '83'],
+?                ^
++    ['height', '130'],
+?                ^ +
+-    ['width', '185']]],
+?                ^^
++    ['width', '120']]],
+?                ^^
+*
+*** Exception: <Level 3> Visual Diff Failure:
+* HTML tag attribute values don't match the baseline!
+```

examples/visual_testing/layout_test.py

@@ -3,16 +3,16 @@
 
 class VisualLayoutTest(BaseCase):
 
-    def test_applitools_helloworld(self):
+    def test_applitools_layout_change(self):
         self.open('https://applitools.com/helloworld?diff1')
-        print('Creating baseline in "visual_baseline" folder...')
+        print('\nCreating baseline in "visual_baseline" folder...')
         self.check_window(name="helloworld", baseline=True)
         self.click('a[href="?diff1"]')
         # Verify html tags match previous version
         self.check_window(name="helloworld", level=1)
-        # Verify html tags + attributes match previous version
+        # Verify html tags and attribute names match previous version
         self.check_window(name="helloworld", level=2)
-        # Verify html tags + attributes + values match previous version
+        # Verify html tags and attribute values match previous version
         self.check_window(name="helloworld", level=3)
         # Change the page enough for a Level-3 comparison to fail
         self.click("button")

examples/visual_testing/python_home_test.py

@@ -0,0 +1,12 @@
+from seleniumbase import BaseCase
+
+
+class VisualLayoutTest(BaseCase):
+
+    def test_python_home_layout_change(self):
+        self.open('https://python.org/')
+        print('\nCreating baseline in "visual_baseline" folder.')
+        self.check_window(name="github_home", baseline=True)
+        # Remove the "Donate" button
+        self.remove_element('a.donate-button')
+        self.check_window(name="github_home", level=0)

examples/visual_testing/test_layout_fail.py

@@ -0,0 +1,30 @@
+from seleniumbase import BaseCase
+
+
+class VisualLayoutFailureTest(BaseCase):
+
+    def test_applitools_layout_change_failure(self):
+        self.open('https://applitools.com/helloworld?diff1')
+        print('\nCreating baseline in "visual_baseline" folder.')
+        self.check_window(name="helloworld", baseline=True)
+        self.click('a[href="?diff1"]')
+        # Change the page enough for a Level-3 comparison to fail
+        self.click("button")
+        self.check_window(name="helloworld", level=3)
+
+    def test_python_home_layout_change_failure(self):
+        self.open('https://python.org/')
+        print('\nCreating baseline in "visual_baseline" folder.')
+        self.check_window(name="python_home", baseline=True)
+        # Remove the "Donate" button
+        self.remove_element('a.donate-button')
+        self.check_window(name="python_home", level=3)
+
+    def test_xkcd_layout_change_failure(self):
+        self.open('https://xkcd.com/554/')
+        print('\nCreating baseline in "visual_baseline" folder.')
+        self.check_window(name="xkcd_554", baseline=True)
+        # Change height: (83 -> 110) , Change width: (185 -> 120)
+        self.set_attribute('[alt="xkcd.com logo"]', "height", "110")
+        self.set_attribute('[alt="xkcd.com logo"]', "width", "120")
+        self.check_window(name="xkcd_554", level=3)

examples/visual_testing/xkcd_visual_test.py

@@ -0,0 +1,13 @@
+from seleniumbase import BaseCase
+
+
+class VisualLayoutTest(BaseCase):
+
+    def test_xkcd_layout_change(self):
+        self.open('https://xkcd.com/554/')
+        print('\nCreating baseline in "visual_baseline" folder.')
+        self.check_window(name="xkcd_554", baseline=True)
+        # Change height: (83 -> 130) , Change width: (185 -> 120)
+        self.set_attribute('[alt="xkcd.com logo"]', "height", "130")
+        self.set_attribute('[alt="xkcd.com logo"]', "width", "120")
+        self.check_window(name="xkcd_554", level=0)

requirements.txt

@@ -2,7 +2,7 @@ pip>=20.0.2
 setuptools>=44.0.0;python_version<"3"
 setuptools>=45.1.0;python_version>="3"
 setuptools-scm>=3.4.3
-wheel>=0.34.1
+wheel>=0.34.2
 six==1.14.0
 nose==1.3.7
 ipdb==0.12.3
@@ -14,7 +14,7 @@ selenium==3.141.0
 pluggy>=0.13.1
 attrs>=19.3.0
 pytest>=4.6.9;python_version<"3"
-pytest>=5.3.4;python_version>="3"
+pytest>=5.3.5;python_version>="3"
 pytest-cov>=2.8.1
 pytest-forked>=1.1.3
 pytest-html==1.22.1;python_version<"3.6"

seleniumbase/core/log_helper.py

@@ -168,8 +168,11 @@ def log_folder_setup(log_path, archive_logs=False):
             archived_folder, int(time.time()))
 
         if len(os.listdir(log_path)) > 0:
-            shutil.move(log_path, archived_logs)
-            os.makedirs(log_path)
+            try:
+                shutil.move(log_path, archived_logs)
+                os.makedirs(log_path)
+            except Exception:
+                pass  # A file was probably open at the time
             if not settings.ARCHIVE_EXISTING_LOGS and not archive_logs:
                 shutil.rmtree(archived_logs)
             else: