[Weibo] add support for Sina Weibo #15079

sprhawk · 2017-12-26T08:48:55Z

Please follow the guide below

You will be asked some questions, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your pull request (like that [x])
Use Preview tab to see how your pull request will actually look like

Before submitting a pull request make sure you have:

At least skimmed through adding new extractor tutorial and youtube-dl coding conventions sections
Searched the bugtracker for similar pull requests
Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

I am the original author of this code and I am willing to release it under Unlicense
I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

Bug fix
Improvement
New extractor
New feature

Description of your pull request and other information

add support for downloading video from Sina Weibo (Weibo.com), both URL from desktop site or mobile site

yan12125

This is good in general, just some comments for minor issues

yan12125 · 2017-12-26T09:43:40Z

youtube_dl/extractor/weibo.py

+            'Accept-Language': 'en,zh-CN;q=0.9,zh;q=0.8',
+            'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36',
+            'Upgrade-Insecure-Requests': '1',
+        }


Are all of these necessary? I think youtube-dl defaults suffice.

yan12125 · 2017-12-26T09:44:23Z

youtube_dl/extractor/weibo.py

+            'Upgrade-Insecure-Requests': '1',
+        }
+        # to get Referer url for genvisitor
+        webpage, urlh = self._download_webpage_handle(url, video_id, headers=headers, note="first visit the page")


Use _ (underline) instead of webpage if the value is not used.

yan12125 · 2017-12-26T09:44:38Z

youtube_dl/extractor/weibo.py

+
+        data = urlencode({
+            "cb": "gen_callback",
+            "fp": '{"os":"2","browser":"Gecko57,0,0,0","fonts":"undefined","screenInfo":"1440*900*24","plugins":""}',


Use json.dumps instead

yan12125 · 2017-12-26T09:45:24Z

youtube_dl/extractor/weibo.py

+            'Referer': visitor_url,
+        }
+
+        r_genvisitor = request.Request(


Use data and headers parameters of _download_webpage instead.

yan12125 · 2017-12-26T09:46:02Z

youtube_dl/extractor/weibo.py

+            data=data,
+            headers=headers,
+        )
+        webpage, urlh = self._download_webpage_handle(r_genvisitor, video_id, note="gen visitor")


Use _download_webpage is urlh is not used. And note should be meaningful for typical users.

yan12125 · 2017-12-26T09:54:46Z

youtube_dl/extractor/weibo.py

+            'Accept-Encoding': 'gzip, deflate, br',
+            'Accept-Language': 'en,zh-CN;q=0.9,zh;q=0.8',
+            'Upgrade-Insecure-Requests': '1',
+        }


yan12125 · 2017-12-26T09:55:08Z

youtube_dl/extractor/weibo.py

+            'Upgrade-Insecure-Requests': '1',
+        }
+        # to get Referer url for genvisitor
+        webpage, urlh = self._download_webpage_handle(url, video_id, headers=headers, note="visit the page")


Same issue for urlh

yan12125 · 2017-12-26T09:56:05Z

youtube_dl/extractor/weibo.py

+            'url': page_info['media_info']['stream_url'],
+            'format': 'mp4',
+        }
+        formats = [format]


You can just use 'url' in the final dict if there's only one format

yan12125 · 2017-12-26T09:56:40Z

youtube_dl/extractor/weibo.py

+            'format': 'mp4',
+        }
+        formats = [format]
+        uploader = weibo_info['status']['user']['screen_name']


This is an optional field; use .get idiom. Check https://github.com/rg3/youtube-dl/blob/master/README.md#mandatory-and-optional-metafields

yan12125 · 2017-12-26T09:56:48Z

youtube_dl/extractor/weibo.py

+            'title': title,
+            'uploader': uploader,
+            'formats': formats
+            # TODO more properties (see youtube_dl/extractor/common.py)


sprhawk · 2017-12-29T05:50:54Z

@yan12125 when can you do a further review over the new changes?

yan12125 · 2017-12-29T09:12:21Z

when can you do a further review over the new changes?

jsonp stuffs are still there, and in some cases _download_webpage_handle can be changed to _download_webpage

sprhawk · 2017-12-29T17:15:04Z

@yan12125 Sorry, missed your reply. Changed them

sprhawk · 2018-01-03T16:22:45Z

@yan12125 I think the travis-ci erros are not related to my code.

yan12125 · 2018-01-04T00:50:39Z

I've restarted the build.

yan12125 · 2018-01-09T10:15:37Z

Thanks for the extractor! Do you want to be listed in AUTHORS? If so under what name?

sprhawk · 2018-01-09T10:23:16Z

Cool, please list me under name: Yang Hongbo Thanks 发自我的 iPhone

…

在 2018年1月9日，18:16，Chih-Hsuan Yen ***@***.***> 写道： Thanks for the extractor! Do you want to be listed in AUTHORS? If so under what name? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

yan12125 · 2018-01-09T11:36:13Z

Done :)

sprhawk added 7 commits December 11, 2017 12:26

simply get the correct webpage, but not parsed to extract information

29ac31a

a working version

3281af3

add other properties; remove print verbose

0c69958

added weibo mobile site support

447a5a7

change to use compat urllib

d2be5bb

re-format code to pass flake8

9510437

fix compat_urllib_request for python2.7

2593651

yan12125 requested changes Dec 26, 2017

View reviewed changes

yan12125 added the pending-fixes label Dec 26, 2017

sprhawk added 2 commits December 26, 2017 22:26

fix according to "ytdl-org/youtube-dl#15079 (comment)"

42a1012

Merge branch 'master' of github.com:rg3/youtube-dl into weibo

c33de00

sprhawk added 3 commits December 30, 2017 01:08

replace urlencode.encode with urlencode_postdata

5c97ec5

replace split with strip_jsonp

6a41a12

replace unused _download_webpage_handle with _download_webpage

48058d8

changed to use .get to get field from json object

6648fd8

yan12125 merged commit 6648fd8 into ytdl-org:master Jan 9, 2018

yan12125 pushed a commit that referenced this pull request Jan 9, 2018

[ChangeLog] Update after #15079

0f71de0

yan12125 pushed a commit that referenced this pull request Jan 9, 2018

Credit @sprhawk for the Weibo extractor (#15079)

5b23845

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Weibo] add support for Sina Weibo #15079

[Weibo] add support for Sina Weibo #15079

sprhawk commented Dec 26, 2017 •

edited

Loading

yan12125 left a comment

yan12125 Dec 26, 2017

yan12125 Dec 26, 2017

yan12125 Dec 26, 2017

yan12125 Dec 26, 2017

yan12125 Dec 26, 2017

yan12125 Dec 26, 2017

yan12125 Dec 26, 2017

yan12125 Dec 26, 2017

yan12125 Dec 26, 2017

yan12125 Dec 26, 2017

sprhawk commented Dec 29, 2017

yan12125 commented Dec 29, 2017

sprhawk commented Dec 29, 2017

sprhawk commented Jan 3, 2018 •

edited

Loading

yan12125 commented Jan 4, 2018

yan12125 commented Jan 9, 2018

sprhawk commented Jan 9, 2018 via email

yan12125 commented Jan 9, 2018

[Weibo] add support for Sina Weibo #15079

[Weibo] add support for Sina Weibo #15079

Conversation

sprhawk commented Dec 26, 2017 • edited Loading

Please follow the guide below

Before submitting a pull request make sure you have:

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

What is the purpose of your pull request?

Description of your pull request and other information

yan12125 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sprhawk commented Dec 29, 2017

yan12125 commented Dec 29, 2017

sprhawk commented Dec 29, 2017

sprhawk commented Jan 3, 2018 • edited Loading

yan12125 commented Jan 4, 2018

yan12125 commented Jan 9, 2018

sprhawk commented Jan 9, 2018 via email

yan12125 commented Jan 9, 2018

sprhawk commented Dec 26, 2017 •

edited

Loading

sprhawk commented Jan 3, 2018 •

edited

Loading