Merge branch 'release-0.2.0'

MANIFEST.in

@@ -4,5 +4,6 @@ include LICENSE
 include MANIFEST.in
 include README.rst
 include mopidy_waitforinternet/ext.conf
+include mopidy_waitfortimesync/ext.conf
 
 recursive-include tests *.py

README.rst

@@ -3,11 +3,11 @@ Mopidy-WaitForInternet
 ****************************
 
 .. image:: https://img.shields.io/pypi/v/Mopidy-WaitForInternet.svg?style=flat
-    :target: https://pypi.python.org/pypi/Mopidy-WaitForInternet/
+    :target: https://pypi.org/project/Mopidy-WaitForInternet/
     :alt: Latest PyPI version
 
 .. image:: https://img.shields.io/pypi/dm/Mopidy-WaitForInternet.svg?style=flat
-    :target: https://pypi.python.org/pypi/Mopidy-WaitForInternet/
+    :target: https://pypi.org/project/Mopidy-WaitForInternet/
     :alt: Number of PyPI downloads
 
 .. image:: https://img.shields.io/github/workflow/status/DavisNT/mopidy-waitforinternet/Python%20build/develop.svg?style=flat
@@ -22,7 +22,7 @@ Mopidy-WaitForInternet
     :target: https://github.com/DavisNT/mopidy-waitforinternet/actions/workflows/servers-test.yml
     :alt: Weekly build that tests connectivity check servers
 
-`Mopidy <http://www.mopidy.com/>`_ extension that waits (up to around 5 minutes) for an internet connection during early phase of Mopidy startup (before other extensions start to initialize).
+`Mopidy <http://www.mopidy.com/>`_ extensions that wait (up to around 5 minutes) for an internet connection (and optionally for time synchronization) during early phase of Mopidy startup (before other extensions start to initialize).
 
 
 Installation
@@ -36,25 +36,57 @@ Install by running::
 Configuration
 =============
 
-This extension has no configuration options in ``mopidy.conf`` apart from the default ``enabled`` setting::
+This package consists of two Mopidy extensions - ``mopidy_waitforinternet`` (enabled by default) that waits **only** for internet connection and ``mopidy_waitfortimesync`` (disabled by default) that waits for internet connection **and** time synchronization. They have no configuration options in ``mopidy.conf`` apart from the default ``enabled`` setting::
 
+    # To enable waiting for internet connection and time synchronization
     [waitforinternet]
-    # To temporary disable this extension without uninstalling it
     enabled = false
 
+    [waitfortimesync]
+    enabled = true
+
+These extensions don't support proxy servers (they ignore proxy configuration in ``mopidy.conf``).
 
 Usage
 =====
 
-This extension will delay initialization of other Mopidy extensions until an internet connection has been initialized (for up to around 5 minutes).
+Mopidy-WaitForInternet might be useful if other Mopidy extensions (e.g. extensions for online music streaming services) fail to initialize, because they try to connect to internet resources before machine running Mopidy has established an internet connection (e.g. connected to wifi) or synchronized its clock.
+
+``mopidy_waitforinternet`` will delay initialization of other Mopidy extensions until an internet connection has been established (the extension will wait for up to around 5 minutes). It's recommended if:
+
+* the computer running Mopidy has a `real-time clock <https://en.wikipedia.org/wiki/Real-time_clock>`_
+
+* all of the below:
+
+  * it is important to minimize Mopidy startup time
+
+  * it is acceptable if other Mopidy extensions occasionally (once in several months or so) fail to initialize due to inaccurate date/time
+
+  * the computer does not have a real-time clock
+
+  * the computer/OS saves the time between reboots (like Raspberry Pi OS does)
+
+  * the computer is used often
 
-Mopidy-WaitForInternet might be useful if other Mopidy extensions (e.g. extensions for online music streaming services) fail to initialize, because they try to connect to internet resources before machine running Mopidy has established an internet connection (e.g. connected to wifi).
+``mopidy_waitfortimesync`` will delay initialization of other Mopidy extensions until an internet connection has been established and computer's clock has been synchronized (the extension will wait for up to around 5 minutes). It's recommended if:
+
+* prolonged Mopidy startup time is not a problem
+
+* it is important to minimize initialization failures of other Mopidy extensions
+
+* the computer running Mopidy does not have a real-time clock and is used rarely
+
+Local time (computer's clock) is somewhat important for connectivity. Most internet services use HTTPS and HTTPS has certificates that are valid for a specific time period (usually 3 or 13 months). To connect to an HTTPS resource, computer's clock must be within the validity period of the HTTPS certificate used by that particular resource. Some computers (e.g. Raspberry Pi) don't have `real-time clocks <https://en.wikipedia.org/wiki/Real-time_clock>`_ and synchronize their clocks from the internet (via `NTP <https://en.wikipedia.org/wiki/Network_Time_Protocol>`_). In most cases, until the clock of such computer is synchronized it is set to the time saved at previous shutdown, for some computers the clock is set to a constant time/date (e.g. midnight January 1, 2020). As ``mopidy_waitforinternet`` uses HTTPS, it will detect internet connectivity only when computer's clock is within the validity period of the HTTPS certificate of at least one of the URLs used by ``mopidy_waitforinternet``. This guarantees that computer's clock has accuracy of a year or so, however this does not guarantee that computer's clock is accurate enough to allow connectivity (to other HTTPS resources) required by other Mopidy extensions.
+
+Both extensions log information about the introduced startup delay.
 
 Important internals
 ===================
 
 Mopidy-WaitForInternet uses several different URLs (currently - requests to public `DoH <https://en.wikipedia.org/wiki/DNS_over_HTTPS>`_ servers) to check internet connectivity. As a future-proofing measure there is a `weekly servers-test build <https://github.com/DavisNT/mopidy-waitforinternet/actions/workflows/servers-test.yml>`_ that verifies availability of these URLs.
 
+Time synchronization is checked by comparing local time with the ``Date`` response header of HTTP requests to the internet connectivity check URLs (difference of less than 10 seconds is considered synchronized time).
+
 License
 =======
 ::
@@ -86,6 +118,12 @@ Project resources
 Changelog
 =========
 
+v0.2.0
+----------------------------------------
+
+- Added second extension (mopidy_waitfortimesync).
+- Minor improvements.
+
 v0.1.1
 ----------------------------------------
 

mopidy_waitforinternet/__init__.py

@@ -1,16 +1,17 @@
 import logging
 import os
 import time
+from datetime import datetime
 
 from mopidy import config, ext
 
 import requests
 
-__version__ = '0.1.1'
+__version__ = '0.2.0'
 
 check_urls = [
     'https://cloudflare-dns.com/dns-query?dns=AAABAAABAAAAAAAACmNsb3VkZmxhcmUDY29tAAABAAE',
-    'https://cloudflare-dns.com/dns-query?dns=AAABAAABAAAAAAAABm1vcGlkeQNjb20AAAEAAQ',
+    'https://security.cloudflare-dns.com/dns-query?dns=AAABAAABAAAAAAAABm1vcGlkeQNjb20AAAEAAQ',
     'https://dns.google/dns-query?dns=AAABAAABAAAAAAAABmdvb2dsZQNjb20AAAEAAQ',
     'https://dns.google/dns-query?dns=AAABAAABAAAAAAAABm1vcGlkeQNjb20AAAEAAQ',
     'https://dns.quad9.net/dns-query?dns=AAABAAABAAAAAAAABm1vcGlkeQNjb20AAAEAAQ'
@@ -29,15 +30,29 @@ def get_default_config(self):
         return config.read(conf_file)
 
     def setup(self, registry):
+        self.__class__.wait_for_internet(False, logger)
+
+    def wait_for_internet(wait_for_timesync, logger):
         retries = 0
+        extramsg = ''
         verified = False
         start = time.monotonic()
         while time.monotonic() - start < 300:
             try:
-                requests.get(check_urls[retries % len(check_urls)], timeout=10, allow_redirects=False)
+                resp = requests.get(check_urls[retries % len(check_urls)], timeout=10, allow_redirects=False)
+                try:
+                    timediff = (datetime.utcnow() - datetime.strptime(resp.headers['Date'], '%a, %d %b %Y %H:%M:%S GMT')).total_seconds()
+                    if wait_for_timesync:
+                        if extramsg == '':
+                            extramsg = ', initial time offset: %ds' % timediff
+                        assert abs(timediff) < 10
+                    extramsg += ', time offset: %ds' % timediff
+                except Exception:
+                    if wait_for_timesync:
+                        raise
                 verified = True
                 break
             except Exception:
                 time.sleep(1)
                 retries += 1
-        logger.info('Internet connectivity verified: %s, retries: %d, time taken %.3fs', verified, retries, time.monotonic() - start)
+        logger.info('Internet connectivity verified: %s, retries: %d, time taken: %.3fs%s', verified, retries, time.monotonic() - start, extramsg)

mopidy_waitfortimesync/__init__.py

@@ -0,0 +1,23 @@
+import logging
+import os
+
+from mopidy import config, ext
+
+import mopidy_waitforinternet
+
+__version__ = mopidy_waitforinternet.__version__
+
+logger = logging.getLogger(__name__)
+
+
+class WaitForTimeSyncExtension(ext.Extension):
+    dist_name = 'Mopidy-WaitForInternet'
+    ext_name = 'waitfortimesync'
+    version = __version__
+
+    def get_default_config(self):
+        conf_file = os.path.join(os.path.dirname(__file__), 'ext.conf')
+        return config.read(conf_file)
+
+    def setup(self, registry):
+        mopidy_waitforinternet.WaitForInternetExtension.wait_for_internet(True, logger)

mopidy_waitfortimesync/ext.conf

@@ -0,0 +1,2 @@
+[waitfortimesync]
+enabled = false

setup.py

@@ -31,6 +31,7 @@ def get_version(filename):
     entry_points={
         'mopidy.ext': [
             'waitforinternet = mopidy_waitforinternet:WaitForInternetExtension',
+            'waitfortimesync = mopidy_waitfortimesync:WaitForTimeSyncExtension',
         ],
     },
     classifiers=[

tests/test_connectivity_check_servers.py

@@ -1,5 +1,6 @@
 import time
 import unittest
+from datetime import datetime
 
 import pytest
 
@@ -21,6 +22,7 @@ def test_servers(self):
         for url in mopidy_waitforinternet.check_urls:
             resp = requests.get(url, timeout=2, allow_redirects=False)
             self.assertEqual(resp.status_code, 200)
+            self.assertLess(abs((datetime.utcnow() - datetime.strptime(resp.headers['Date'], '%a, %d %b %Y %H:%M:%S GMT')).total_seconds()), 5)
         t_stop = time.monotonic()
 
         self.assertLess(t_stop - t_start, len(mopidy_waitforinternet.check_urls))

tests/test_server_time.py

@@ -0,0 +1,142 @@
+import time
+import unittest
+
+from freezegun import freeze_time
+
+import mock
+
+import mopidy_waitfortimesync
+
+import responses
+
+import mopidy_waitforinternet
+
+
+@responses.activate
+def TestDateHeaders(test, ext):
+    responses.add(responses.GET, 'https://nosuchhost.nosuchdomain.arpa/date-header-invalid', headers={'Date': 'not a date'})
+    responses.add(responses.GET, 'https://nosuchhost.nosuchdomain.arpa/date-header-2022-07-01-14-00-20', headers={'Date': 'Fri, 01 Jul 2022 14:00:20 GMT'})
+    responses.add(responses.GET, 'https://nosuchhost.nosuchdomain.arpa/date-header-2022-07-01-14-00-30', headers={'Date': 'Fri, 01 Jul 2022 14:00:30 GMT'})
+    responses.add(responses.GET, 'https://nosuchhost.nosuchdomain.arpa/date-header-2022-07-01-14-00-40', headers={'Date': 'Fri, 01 Jul 2022 14:00:40 GMT'})
+    responses.add(responses.GET, 'https://nosuchhost.nosuchdomain.arpa/date-header-2022-07-01-14-00-50', headers={'Date': 'Fri, 01 Jul 2022 14:00:50 GMT'})
+    registry = mock.Mock()
+
+    t_start = time.monotonic()
+    ext.setup(registry)
+    t_stop = time.monotonic()
+
+    registry.add.assert_not_called()
+    return t_stop - t_start
+
+
+class TestsWithInvalidDateHeader(unittest.TestCase):
+
+    def setUp(self):
+        self.backup_check_urls = mopidy_waitforinternet.check_urls
+        mopidy_waitforinternet.check_urls = [
+            'https://nosuchhost.nosuchdomain.arpa/date-header-invalid',
+            'https://nosuchhost.nosuchdomain.arpa/date-header-2022-07-01-14-00-20',
+            'https://nosuchhost.nosuchdomain.arpa/date-header-2022-07-01-14-00-30',
+            'https://nosuchhost.nosuchdomain.arpa/date-header-2022-07-01-14-00-40',
+            'https://nosuchhost.nosuchdomain.arpa/date-header-2022-07-01-14-00-50'
+        ]
+
+    def test_waitforinternet_setup_complex_actual_date(self):
+        duration = TestDateHeaders(self, mopidy_waitforinternet.WaitForInternetExtension())
+        self.assertGreaterEqual(duration, 0)
+        self.assertLess(duration, 0.4)
+
+    @freeze_time('2022-07-01 14:00:00', ignore=['tests'])
+    def test_waitforinternet_setup_complex_at140000(self):
+        duration = TestDateHeaders(self, mopidy_waitforinternet.WaitForInternetExtension())
+        self.assertGreaterEqual(duration, 0)
+        self.assertLess(duration, 0.4)
+
+    @freeze_time('2022-07-01 14:00:20', ignore=['tests'])
+    def test_waitforinternet_setup_complex_at140020(self):
+        duration = TestDateHeaders(self, mopidy_waitforinternet.WaitForInternetExtension())
+        self.assertGreaterEqual(duration, 0)
+        self.assertLess(duration, 0.4)
+
+    @freeze_time('2022-07-01 18:18:20', ignore=['tests'])
+    def test_waitforinternet_setup_complex_at181820(self):
+        duration = TestDateHeaders(self, mopidy_waitforinternet.WaitForInternetExtension())
+        self.assertGreaterEqual(duration, 0)
+        self.assertLess(duration, 0.4)
+
+    @freeze_time('2022-07-01 14:00:20', ignore=['tests'])
+    def test_waitfortimesync_setup_complex_at140020(self):
+        duration = TestDateHeaders(self, mopidy_waitfortimesync.WaitForTimeSyncExtension())
+        self.assertGreaterEqual(duration, 1)
+        self.assertLess(duration, 1.4)
+
+    @freeze_time('2022-07-01 14:00:35', ignore=['tests'])
+    def test_waitfortimesync_setup_complex_at140035(self):
+        duration = TestDateHeaders(self, mopidy_waitfortimesync.WaitForTimeSyncExtension())
+        self.assertGreaterEqual(duration, 2)
+        self.assertLess(duration, 2.4)
+
+    @freeze_time('2022-07-01 14:00:59', ignore=['tests'])
+    def test_waitfortimesync_setup_complex_at140059(self):
+        duration = TestDateHeaders(self, mopidy_waitfortimesync.WaitForTimeSyncExtension())
+        self.assertGreaterEqual(duration, 4)
+        self.assertLess(duration, 4.4)
+
+    def tearDown(self):
+        mopidy_waitforinternet.check_urls = self.backup_check_urls
+
+
+class TestsNoInvalidDateHeader(unittest.TestCase):
+
+    def setUp(self):
+        self.backup_check_urls = mopidy_waitforinternet.check_urls
+        mopidy_waitforinternet.check_urls = [
+            'https://nosuchhost.nosuchdomain.arpa/date-header-2022-07-01-14-00-20',
+            'https://nosuchhost.nosuchdomain.arpa/date-header-2022-07-01-14-00-30',
+            'https://nosuchhost.nosuchdomain.arpa/date-header-2022-07-01-14-00-40',
+            'https://nosuchhost.nosuchdomain.arpa/date-header-2022-07-01-14-00-50'
+        ]
+
+    def test_waitforinternet_setup_complex_actual_date(self):
+        duration = TestDateHeaders(self, mopidy_waitforinternet.WaitForInternetExtension())
+        self.assertGreaterEqual(duration, 0)
+        self.assertLess(duration, 0.4)
+
+    @freeze_time('2022-07-01 14:00:00', ignore=['tests'])
+    def test_waitforinternet_setup_complex_at140000(self):
+        duration = TestDateHeaders(self, mopidy_waitforinternet.WaitForInternetExtension())
+        self.assertGreaterEqual(duration, 0)
+        self.assertLess(duration, 0.4)
+
+    @freeze_time('2022-07-01 14:00:20', ignore=['tests'])
+    def test_waitforinternet_setup_complex_at140020(self):
+        duration = TestDateHeaders(self, mopidy_waitforinternet.WaitForInternetExtension())
+        self.assertGreaterEqual(duration, 0)
+        self.assertLess(duration, 0.4)
+
+    @freeze_time('2022-07-01 18:18:20', ignore=['tests'])
+    def test_waitforinternet_setup_complex_at181820(self):
+        duration = TestDateHeaders(self, mopidy_waitforinternet.WaitForInternetExtension())
+        self.assertGreaterEqual(duration, 0)
+        self.assertLess(duration, 0.4)
+
+    @freeze_time('2022-07-01 14:00:20', ignore=['tests'])
+    def test_waitfortimesync_setup_complex_at140020(self):
+        duration = TestDateHeaders(self, mopidy_waitfortimesync.WaitForTimeSyncExtension())
+        self.assertGreaterEqual(duration, 0)
+        self.assertLess(duration, 0.4)
+
+    @freeze_time('2022-07-01 14:00:35', ignore=['tests'])
+    def test_waitfortimesync_setup_complex_at140035(self):
+        duration = TestDateHeaders(self, mopidy_waitfortimesync.WaitForTimeSyncExtension())
+        self.assertGreaterEqual(duration, 1)
+        self.assertLess(duration, 1.4)
+
+    @freeze_time('2022-07-01 14:00:59', ignore=['tests'])
+    def test_waitfortimesync_setup_complex_at140059(self):
+        duration = TestDateHeaders(self, mopidy_waitfortimesync.WaitForTimeSyncExtension())
+        self.assertGreaterEqual(duration, 3)
+        self.assertLess(duration, 3.4)
+
+    def tearDown(self):
+        mopidy_waitforinternet.check_urls = self.backup_check_urls

tests/test_extension.py → tests/test_waitforinternet.py

@@ -1,3 +1,4 @@
+import socket
 import time
 import unittest
 
@@ -8,6 +9,9 @@
 
 class WaitForInternetExtensionTest(unittest.TestCase):
 
+    def setUp(self):
+        self.backup_check_urls = mopidy_waitforinternet.check_urls
+
     def test01_get_default_config(self):
         ext = mopidy_waitforinternet.WaitForInternetExtension()
 
@@ -27,9 +31,21 @@ def test02_setup_realrun(self):  # DO NOT MODIFY mopidy_waitforinternet.check_ur
 
         registry.add.assert_not_called()
         self.assertGreater(t_stop - t_start, 0)
-        self.assertLess(t_stop - t_start, 2)
+        self.assertLess(t_stop - t_start, 0.999)
 
     def test03_setup_transient_nameresolutionfailure(self):
+        for name in [
+            'nosuchhost.nosuchdomain1.arpa',
+            'nosuchhost.nosuchdomain2.arpa',
+            'nosuchhost.nosuchdomain3.arpa',
+            'nosuchhost.nosuchdomain4.arpa',
+            'nosuchhost.nosuchdomain5.arpa'
+        ]:
+            try:
+                socket.gethostbyname(name)
+            except Exception:
+                pass
+
         registry = mock.Mock()
 
         mopidy_waitforinternet.check_urls = [
@@ -95,3 +111,6 @@ def test04_setup_permanent_nameresolutionfailure(self):
         registry.add.assert_not_called()
         self.assertGreaterEqual(t_stop - t_start, 300)
         self.assertLess(t_stop - t_start, 305)
+
+    def tearDown(self):
+        mopidy_waitforinternet.check_urls = self.backup_check_urls