Nest Add new extractor #31274

evanzh15 · 2022-10-02T16:49:30Z

Please follow the guide below

You will be asked some questions, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your pull request (like that [x])
Use Preview tab to see how your pull request will actually look like

Before submitting a pull request make sure you have:

Searched the bugtracker for similar pull requests
Read adding new extractor tutorial
Read youtube-dl coding conventions and adjusted the code to meet them
Covered the code with tests (note that PRs without tests will be REJECTED)
Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

I am the original author of this code and I am willing to release it under Unlicense
I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

Bug fix
Improvement
New extractorExplanation of your pull request in arbitrary form goes here. Please make sure the description explains the purpose and effect of your pull request and is worded well enough to be understood. Provide as much context and examples as possible.
New feature

Description of your pull request and other information

Add extractor for NestCam video.

dirkf

Thanks for your work!

It's nearly there. Have a look at the suggestions and get the test working.

dirkf · 2022-10-10T18:32:38Z

youtube_dl/extractor/nest.py

+            r'https:\/\/video.nest.com\/clip\/(.+?)(\.|")', webpage, 'video_id', fatal=False)
+        title = self._html_search_meta(['og:title', 'title'], webpage, 'title')
+        if title == "":
+            title = "\"\""


In this extractor the page may have no explicit title, but yt-dl wants one, so use a specialised standard method to invent one (as above):

Suggested change

title = "\"\""

title = self._generic_title(url)

dirkf · 2022-10-10T18:35:10Z

youtube_dl/extractor/nest.py

+            'description': '#caughtonNestCam',
+        }
+    }
+


Suggested change

def _generic_title(self, url)

return 'NestCam video ' + super(NestIE, self)._generic_title(url)

dirkf · 2022-10-10T18:36:01Z

youtube_dl/extractor/nest.py

+        video_id = self._search_regex(
+            r'https:\/\/video.nest.com\/clip\/(.+?)(\.|")', webpage, 'video_id', fatal=False)
+        title = self._html_search_meta(['og:title', 'title'], webpage, 'title')
+        if title == "":


Suggested change

if title == "":

if not title:

dirkf · 2022-10-10T18:42:13Z

youtube_dl/extractor/nest.py

+        webpage = self._download_webpage(url, video_id)
+        video_id = self._search_regex(
+            r'https:\/\/video.nest.com\/clip\/(.+?)(\.|")', webpage, 'video_id', fatal=False)
+        title = self._html_search_meta(['og:title', 'title'], webpage, 'title')


Prefer tuple for const sequence:

Suggested change

title = self._html_search_meta(['og:title', 'title'], webpage, 'title')

title = self._html_search_meta(('og:title', 'title'), webpage, 'title')

dirkf · 2022-10-10T18:57:50Z

youtube_dl/extractor/nest.py

+        if "/" in ext:
+            ext = ext[ext.index("/") + 1:]


Use utils.mimetype2ext():

Suggested change

if "/" in ext:

ext = ext[ext.index("/") + 1:]

ext = mimetype2ext(ext) or ext

dirkf · 2022-10-10T19:10:45Z

youtube_dl/extractor/nest.py

+# coding: utf-8
+from __future__ import unicode_literals
+
+from .common import InfoExtractor


Used later:

Suggested change

from .common import InfoExtractor

from .common import InfoExtractor

from ..utils import (

ExtractorError,

mimetype2ext,

url_or_none,

)

dirkf · 2022-10-10T19:21:57Z

youtube_dl/extractor/nest.py

+
+
+class NestIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?video.nest\.com/clip/(?P<id>)(.mp4)?'


This will never match a useful ID!

Suggested change

_VALID_URL = r'https?://(?:www\.)?video.nest\.com/clip/(?P<id>)(.mp4)?'

_VALID_URL = r'https?://(?:www\.)?video\.nest\.com/clip/(?P<id>\w+)'

dirkf · 2022-10-10T19:24:42Z

youtube_dl/extractor/nest.py

+        video_id = self._search_regex(
+            r'https:\/\/video.nest.com\/clip\/(.+?)(\.|")', webpage, 'video_id', fatal=False)


No need to escape / here (in a JS /regexp/, yes), but do escape ., and don't overwrite video_id:

Suggested change

video_id = self._search_regex(

r'https:\/\/video.nest.com\/clip\/(.+?)(\.|")', webpage, 'video_id', fatal=False)

video_id = self._search_regex(

r'https://video\.nest\.com/clip/(.+?)(?:\.|")', webpage, 'video_id', fatal=False) or video_id

Actually, is this ever different from the value extracted from the page URL? With the correct _VALID_URL, you should have a good value for it. If you do need to do this search, use the _VALID_URL again:

Suggested change

video_id = self._search_regex(

r'https:\/\/video.nest.com\/clip\/(.+?)(\.|")', webpage, 'video_id', fatal=False)

video_id = self._search_regex(

self._VALID_URL, webpage, 'video_id', group='id', fatal=False) or video_id

Or just

Suggested change

video_id = self._search_regex(

r'https:\/\/video.nest.com\/clip\/(.+?)(\.|")', webpage, 'video_id', fatal=False)

dirkf · 2022-10-10T19:29:16Z

youtube_dl/extractor/nest.py

+    _TEST = {
+        'url': 'https://video.nest.com/clip/73ddb6bd57c4485597a76e154a4429ea.mp4',
+        'md5': '7ab4eb6d4c2480be1740cc014a76ee96',
+        'info_dict': {
+            'id': '73ddb6bd57c4485597a76e154a4429ea',
+            'ext': 'mp4',
+            'title': "\"\"",
+            'description': '#caughtonNestCam',
+        }
+    }


Prefer _TESTS_ in new extractors:

Suggested change

_TEST = {

'url': 'https://video.nest.com/clip/73ddb6bd57c4485597a76e154a4429ea.mp4',

'md5': '7ab4eb6d4c2480be1740cc014a76ee96',

'info_dict': {

'id': '73ddb6bd57c4485597a76e154a4429ea',

'ext': 'mp4',

'title': "\"\"",

'description': '#caughtonNestCam',

}

}

_TESTS = [{

'url': 'https://video.nest.com/clip/73ddb6bd57c4485597a76e154a4429ea.mp4',

'md5': '7ab4eb6d4c2480be1740cc014a76ee96',

'info_dict': {

'id': '73ddb6bd57c4485597a76e154a4429ea',

'ext': 'mp4',

'title': "\"\"",

'description': '#caughtonNestCam',

}

}]

dirkf · 2022-10-10T19:31:21Z

youtube_dl/extractor/nest.py

+        'info_dict': {
+            'id': '73ddb6bd57c4485597a76e154a4429ea',
+            'ext': 'mp4',
+            'title': "\"\"",


To match other changes:

Suggested change

'title': "\"\"",

'title': r're:^NestCam video \w+',

Nest Add new extractor

149e69e

evanzh15 mentioned this pull request Oct 2, 2022

Add support for clips from Nest Cam site video.nest.com #31163

Open

5 tasks

Fixes ytdl-org#31163

6dcbe07

dirkf linked an issue Oct 10, 2022 that may be closed by this pull request

Add support for clips from Nest Cam site video.nest.com #31163

Open

5 tasks

dirkf requested changes Oct 10, 2022

View reviewed changes

dirkf mentioned this pull request Dec 2, 2023

Added changes to add nest extractor #32616

Draft

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nest Add new extractor #31274

Nest Add new extractor #31274

evanzh15 commented Oct 2, 2022 •

edited by dirkf

dirkf left a comment

dirkf Oct 10, 2022

dirkf Oct 10, 2022

dirkf Oct 10, 2022

dirkf Oct 10, 2022

dirkf Oct 10, 2022

dirkf Oct 10, 2022

dirkf Oct 10, 2022

dirkf Oct 10, 2022

dirkf Oct 10, 2022

dirkf Oct 10, 2022



	def _generic_title(self, url)
	return 'NestCam video ' + super(NestIE, self)._generic_title(url)

	title = self._html_search_meta(['og:title', 'title'], webpage, 'title')
	title = self._html_search_meta(('og:title', 'title'), webpage, 'title')

	if "/" in ext:
	ext = ext[ext.index("/") + 1:]
	ext = mimetype2ext(ext) or ext

-from .common import InfoExtractor
+from .common import InfoExtractor
+from ..utils import (
+    ExtractorError,
+    mimetype2ext,
+    url_or_none,
+)



		class NestIE(InfoExtractor):
		_VALID_URL = r'https?://(?:www\.)?video.nest\.com/clip/(?P<id>)(.mp4)?'

	_VALID_URL = r'https?://(?:www\.)?video.nest\.com/clip/(?P<id>)(.mp4)?'
	_VALID_URL = r'https?://(?:www\.)?video\.nest\.com/clip/(?P<id>\w+)'

		video_id = self._search_regex(
		r'https:\/\/video.nest.com\/clip\/(.+?)(\.\|")', webpage, 'video_id', fatal=False)

Nest Add new extractor #31274

Are you sure you want to change the base?

Nest Add new extractor #31274

Conversation

evanzh15 commented Oct 2, 2022 • edited by dirkf

Please follow the guide below

Before submitting a pull request make sure you have:

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

What is the purpose of your pull request?

Description of your pull request and other information

dirkf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evanzh15 commented Oct 2, 2022 •

edited by dirkf