Add Support For Base Station Media Downloads with RATLS

[phene]

Follow-up to #86 and based on work done here by @jesserockz

This adds some new classes:

• ArloRatls - Triggers base station port opening, token issuance, and client TLS against the base station.
• ArloBaseStationMediaLibrary - Subclass of ArloMediaLibrary which can iterate over the base station library and trigger downloads of videos
• SecurityUtils - A near-exact copy of @jesserockz 's library for managing the creation of certificates and management of certificates signed for each device.

Example usage:

  for station in arlo.base_stations:
    print(f"Connecting to base station {station.name}: {station.device_id}")
    station.build_ratls()

    print(f"Downloading media from {station.name}...")
    station.build_media_library()

    time.sleep(2)
    while(len(station.ml._downloader._queue) > 0):
      time.sleep(2)

[jesserockz]

OOOHHHH, Exciting!!

I will give this a try.

[Jeff-Rand]

@Jeff-Rand are there any days in which you don't have videos within the library_days period? I've pushed a fix for that specific error, but I'm concerned about the details of that URLError.

Consider adding self._arlo.warning(f"Downloading {self.video_url} to {filename}...") to the top of both download_video methods in media.py. I wonder if you have non-2k videos on your base station, which is causing it to load the wrong object. If you can send me some some of the output, I could probably mitigate it.

Also, can we move this discussion to the PR?

Yes, I currently only have files back to January 1st and I believe the default period is 30 days. When I run it now, the URLError seems to repeat endlessly now. I had to ctrl-C to stop. I will set the period shorter to see what happens.

And yes, I do have 4K videos on my base station. I will add the extra code to media.py and let you know the result.

[phene]

@Jeff-Rand I've pushed another update to better support both 2k and 4k video from the base station.

[Jeff-Rand]

I set the library_days to 2 and now I get request-error=URLError repeated three times. I also added self._arlo.warning(f"Downloading {self.video_url} to {filename}...") to the media.py under the def download_video but I am not getting any output from it.

[Jeff-Rand]

Confirmed that request-error=URLError repeats for library_days+1. But, I haven't been able to pin down where the error is coming from. I'm not sure we are getting to download_video. I will try to add more messages to see where it is coming from.

[phene]

I presume those errors are from trying to list the contents of the library. You might check

1. print(arlo.base_stations[0].ratls.url)
2. print(arlo.be.get(f"/hmsweb/users/device/ratls/status/{arlo.base_stations[0].device_id}"))

[phene]

FYI, the reason you see it library_days + 1 times is because it checks the previous library_days days plus today.

[phene]

Also, are you on the same local network as your base station?

[Jeff-Rand]

Adding the two print statements I get:
https://192.168.0.30:19435
{'ratlsEnabled': True, 'remoteAccessEnabled': True}

[Jeff-Rand]

Also, are you on the same local network as your base station?

Yes

[phene]

I think I just spotted the issue... Update line 51 of ratls:

diff --git a/pyaarlo/ratls.py b/pyaarlo/ratls.py
index ab3a584..84112a4 100644
--- a/pyaarlo/ratls.py
+++ b/pyaarlo/ratls.py
@@ -48,7 +48,7 @@ class ArloRatls(object):
         return self._base_connection_details

     def get(self, path, raw=False):
-        request = Request(f"{self.url}/{path}")
+        request = Request(f"{self.url}{path}")
         request.get_method = lambda: 'GET'

         for (k, v) in self._ratls_req_headers().items():

[Jeff-Rand]

I removed the slash and the result is the same.

[Jeff-Rand]

I added a line to output the request. This is what I got:
Downloading media from Arlo Base...
https://192.168.0.30:19435/hmsls/list/20220126/20220126
request-error=URLError
https://192.168.0.30:19435/hmsls/list/20220127/20220127
request-error=URLError

[phene]

Consider:

diff --git a/pyaarlo/ratls.py b/pyaarlo/ratls.py
index 84112a4..ab03083 100644
--- a/pyaarlo/ratls.py
+++ b/pyaarlo/ratls.py
@@ -60,7 +60,7 @@ class ArloRatls(object):
                 return response
             return json.loads(response.read())
         except Exception as e:
-            self._arlo.warning("request-error={}".format(type(e).__name__))
+            self._arlo.warning("request-error={}:{}".format(type(e).__name__, e.reason))
             return None

     def _ratls_req_headers(self):

I'm struggling to imagine how those URLs would cause URLError exceptions.

[Jeff-Rand]

The error message is now request-error=URLError:Cannot create a client socket with a PROTOCOL_TLS_SERVER context (_ssl.c:801)

[phene]

What version of python are you running? I think something may have changed with Python 3.10's SSL Contexts... pydicom/pynetdicom#709

[Jeff-Rand]

Ahhh.... I'm running Python 3.10.2

Let me try an older version.

[phene]

I need to confirm, but I would bet the base station doesn't support TLS1.2. I can try explicitly allowing it TLS1.1

[Jeff-Rand]

I tried it on another system I have running Python 3.9.7. No errors, yay!

[Jeff-Rand]

Another silly question, what is the default path that the video files are saved to? How do I change the path? Sorry for all the questions, this is all still new to me.

[phene]

It's the original save_media_to configuration option.

Try this diff with python 3.10:

diff --git a/pyaarlo/ratls.py b/pyaarlo/ratls.py
index 84112a4..7414abb 100644
--- a/pyaarlo/ratls.py
+++ b/pyaarlo/ratls.py
@@ -91,6 +91,7 @@ class ArloRatls(object):
         device_certs_path = self._security.device_certs_path(self._unique_id)
         self._sslcontext = ssl.create_default_context(cafile=os.path.join(certs_path, "ica.crt"), purpose=ssl.Purpose.CLIENT_AUTH) #, purp
         self._sslcontext.load_cert_chain(os.path.join(device_certs_path, "peer.crt"), self._security.private_key_path)
+        self._sslcontext.options &= ~ssl.OP_NO_TLSv1_1
         self._base_client = build_opener(HTTPSHandler(context=self._sslcontext))

     def _check_device_certs(self):

[Jeff-Rand]

I'm getting the same error message after adding the self._sslcontext.options &= ~ssl.OP_NO_TLSv1_1

[Jeff-Rand]

I finally figured out the save_media_to option and videos are downloading. Yay,! Only one issue, I am only getting video files from one of my 4K cameras. The videos from the other 4K camera and my doorbell camera don't appear to be in the queue.

If I enumerate my cameras with this code:
for camera in arlo.cameras: print("camera: name={},device_id={},state={}".format(camera.name,camera.device_id,camera.state))

I get this:
camera: name=Front door,device_id=A2G304K0A17D4,state=idle
camera: name=Driveway,device_id=A431087AA236E,state=idle
camera: name=Front yard,device_id=A431087DA2A16,state=idle

I am only getting videos from the last camera named "Front yard"

[phene]

Looks like it only supports TLS 1.0 and SSLv3. Try this option: self._sslcontext.options &= ~ssl.OP_NO_TLSv1

[phene]

This would iterate over the cameras attached to your base station and give you only results for each camera. I haven't found any evidence from decompiling and examining the Android app that the API supports pagination

diff --git a/pyaarlo/media.py b/pyaarlo/media.py
index afa9212..5e3701d 100644
--- a/pyaarlo/media.py
+++ b/pyaarlo/media.py
@@ -311,10 +311,12 @@ class ArloBaseStationMediaLibrary(ArloMediaLibrary):

         # Fetch each page individually, since the base station still only return results for one date at a time
         for date in range(int(date_from), int(date_to) + 1):
-            # This URL is mysterious -- it won't return multiple days of videos
-            data = self._base.ratls.get(f"{RATLS_LIBRARY_PATH}/{date}/{date}")
-            if data and "data" in data:
-                list += data["data"]
+            for camera in self._arlo.cameras:
+                if camera.parent_id == self._base.device_id:
+                    # This URL is mysterious -- it won't return multiple days of videos
+                    data = self._base.ratls.get(f"{RATLS_LIBRARY_PATH}/{date}/{date}/{camera.device_id}")
+                    if data and "data" in data:
+                        list += data["data"]

[Jeff-Rand]

Using self._sslcontext.options &= ~ssl.OP_NO_TLSv1 I still get the error using Python 3.10

[phene]

I'll have to upgrade and try to fix it with Python 3.10 myself. Will push a fix once I find one.

[phene]

@Jeff-Rand the latest change properly supports Python 3.10 (also tested with Python 3.9)

I don't think it was the deprecation of TLS 1.1, but changes related to hostname verification.

[Jeff-Rand]

I'm still getting errors on Python 3.10. The error has changed to request-error=HTTPError. Specifically, the error is HTTPError:Not Found.

Also, on Python 3.9 I'm still only getting videos from one camera. I'm starting to get an understanding of the code and should be able to figure out the issue soon. I will let you know what I find.

[phene]

I had a strange issue with my base station today and I was getting that error until I power cycled it and tried again after a few minutes.

[Jeff-Rand]

I rebooted my base station and tried again with Python 3.10. I'm still getting the not found HTTPError.

I did however fix the problem with getting videos from only one of my three cameras. I made the following changes and it is getting them all now.

In media.py I changed lines 192 and 261 from this:
if content_type.startswith("video/") or content_type == "2k":
to this:
if content_type.startswith("video/") or content_type == "2k" or content_type == "4k" or content_type == "hd":

And line 389 from this:
if self.content_type == '2k' or self.content_type == '4k':
to this:
if self.content_type == '2k' or self.content_type == '4k' or self.content_type == 'hd':

Thanks so much for doing all this work! I am very happy now.

[phene]

@Jeff-Rand cool -- added support for all 3 video content types.

[twrecked]

Firstly, thanks everybody for doing/helping with this.

Sorry for the delay but I'll take a look at all this this weekend.

[Jeff-Rand]

@phene The latest update works great in Python 3.9. I'm now getting the Exception: Failed to gain connectivity to ratls! in Python 3.10

You can ignore the part about it not working in Python 3.10. Apparently, the issue with connectivity has something to do with my machine. I just upgraded my other machine to Python 3.10 and it works just fine.

[phene]

Rebased against master

[twrecked]

Code looks good to me. I'll merge over today.

@phene Thanks for doing this. You still ok with the merge?

[phene]

Go for it!

[twrecked]

Done. And apologies for the delay, I finally for a day off my real job!