Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trying to add new IE #869

Closed
yasoob opened this issue Jun 5, 2013 · 4 comments
Closed

Trying to add new IE #869

yasoob opened this issue Jun 5, 2013 · 4 comments

Comments

@yasoob
Copy link
Contributor

@yasoob yasoob commented Jun 5, 2013

I am trying to add an IE for hypem.com but i am facing some problems. Here is my code (Its test code)
'''
class HypemIE(InfoExtractor):
"""Information Extractor for hypem"""
_VALID_URL = r'(?:http://)?(?:www.)?hypem.com/track/([^/]+)/([^/]+)'

def removeDisallowedFilenameChars(filename):
    validFilenameChars = "-_.() %s%s" % (string.ascii_letters, string.digits)
    cleanedFilename = unicodedata.normalize('NFKD', filename).encode('ASCII', 'ignore')
    return ''.join(c for c in cleanedFilename if c in validFilenameChars)

def _real_extract(self,url):
    mobj = re.match(self._VALID_URL, url)
    if mobj is None:
        raise ExtractorError(u'Invalid URL: %s' % url)
    data = {'ax':1 ,
              'ts': time()
          }
    data_encoded = urllib.urlencode(data)
    complete_url = url + "?"+data_encoded
    request = urllib2.Request(complete_url)
    response = urllib2.urlopen(request)
    #save our cookie
    cookie = response.headers.get('Set-Cookie')
    #grab the HTML
    html = response.read()
    response.close()
    track_list = []
    list_data = re.search(r'<script type="application/json" id="displayList-data">\n    (.*)    </script>',html)
    html_tracks = list_data.group(1)
    if html_tracks is None:
      tracks = track_list
    try:
      track_list = json.loads(html_tracks)
      tracks = track_list[u'tracks']
    except ValueError:
      print "Hypemachine contained invalid JSON."
      tracks =  track_list

    for track in tracks:
        key = track[u"key"]
        id = track[u"id"]
        artist = removeDisallowedFilenameChars(track[u"artist"])
        title = removeDisallowedFilenameChars(track[u"song"])
        type = track[u"type"]
    if type is False:
        continue
    serve_url = "http://hypem.com/serve/source/{}/{}".format(id, key)
    request = urllib2.Request(serve_url, "" , {'Content-Type': 'application/json'})
    request.add_header('cookie', cookie)
    response = urllib2.urlopen(request)
    song_data_json = response.read()
    response.close()
    song_data = json.loads(song_data_json)
    final_url = song_data[u"url"]
    return [{
        'id':       id,
        'url':      final_url,
        'ext':      "mp3",
        'title':    title,
        'artist':   artist,
    }]

'''
and i also added "HypemIE()" in "gen_extractors()" . The problem is that whenever i give it a hypem url it falls back to genericIE instead of using HypemIE. The input which i gave is:

python youtube_dl/main.py http://hypem.com/track/1q6dr/Rone+-+Parade
The traceback for the error is:
'''
Traceback (most recent call last):
File "youtube_dl/main.py", line 18, in
youtube_dl.main()
File "/usr/local/lib/python2.6/dist-packages/youtube_dl-2013.03.29-py2.6.egg/youtube_dl/init.py", line 540, in main
_real_main()
File "/usr/local/lib/python2.6/dist-packages/youtube_dl-2013.03.29-py2.6.egg/youtube_dl/init.py", line 524, in _real_main
retcode = fd.download(all_urls)
File "/usr/local/lib/python2.6/dist-packages/youtube_dl-2013.03.29-py2.6.egg/youtube_dl/FileDownloader.py", line 569, in download
videos = ie.extract(url)
File "/usr/local/lib/python2.6/dist-packages/youtube_dl-2013.03.29-py2.6.egg/youtube_dl/InfoExtractors.py", line 96, in extract
return self._real_extract(url)
File "/usr/local/lib/python2.6/dist-packages/youtube_dl-2013.03.29-py2.6.egg/youtube_dl/InfoExtractors.py", line 1389, in _real_extract
if self._test_redirect(url): return
File "/usr/local/lib/python2.6/dist-packages/youtube_dl-2013.03.29-py2.6.egg/youtube_dl/InfoExtractors.py", line 1378, in _test_redirect
response = opener.open(HeadRequest(url))
File "/usr/lib/python2.6/urllib2.py", line 397, in open
response = meth(req, response)
File "/usr/lib/python2.6/urllib2.py", line 510, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.6/urllib2.py", line 429, in error
result = self._call_chain(_args)
File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain
result = func(_args)
File "/usr/lib/python2.6/urllib2.py", line 616, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "/usr/lib/python2.6/urllib2.py", line 397, in open
response = meth(req, response)
File "/usr/lib/python2.6/urllib2.py", line 510, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.6/urllib2.py", line 429, in error
result = self._call_chain(_args)
File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain
result = func(_args)
File "/usr/lib/python2.6/urllib2.py", line 616, in http_error_302
return self.parent.open(new, timeout=req.timeout)
File "/usr/lib/python2.6/urllib2.py", line 397, in open
response = meth(req, response)
File "/usr/lib/python2.6/urllib2.py", line 510, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.6/urllib2.py", line 435, in error
return self._call_chain(_args)
File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain
result = func(_args)
File "/usr/lib/python2.6/urllib2.py", line 518, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 404: Not Found
'''

I hope that you guys can help me. Any useful comments on my IE code are also welcome.

@jaimeMF
Copy link
Collaborator

@jaimeMF jaimeMF commented Jun 5, 2013

Did you add HypemIE() before genericIE?

@jaimeMF
Copy link
Collaborator

@jaimeMF jaimeMF commented Jun 5, 2013

If you want further help, you can open a branch in your fork so that we can comment and test the code.

@yasoob
Copy link
Contributor Author

@yasoob yasoob commented Jun 5, 2013

@jaimeMF Heres my fork in which i added hypem https://github.com/yasoob/youtube-dl i hope you can test it and provide me with some help.

@yasoob
Copy link
Contributor Author

@yasoob yasoob commented Jun 5, 2013

@jaimeMF jaimeMF closed this Jun 5, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.