Unable to parse course videos from JSON

[rafiulnakib]

Hi,

I was hoping you could help me with this. I followed your method and could download videos of one course. However, while trying to download videos of another course, I got the following error.

Can you please show me any work around?

Thanks.

[soraxas]

Can you post the run with the debug flag and post the log file somewhere? aka. run with

./run.sh URL --debug

Thanks!

[rafiulnakib]

I just did, same thing. :/

[soraxas]

Can you post the log file that generates in the same folder for debugging? It was the purpose for my last comment, thanks!

[rafiulnakib]

Sorry my bad, I didn't understand what you meant and I still don't know where to look for it. I could find one debug.log for pip, not sure if that's the one. I am not very proficient in these matters so if you could help a bit that would be awesome! Thanks.

[soraxas]

No worries!
It should be named as echo360Downloader.log in the same folder as the run.sh script.

Like the following:

You can upload to any sort of filehosting website or https://pastebin.com/

[rafiulnakib]

[DEBUG: 12-04 19:13] main binary_downloader link: ('https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-macosx.zip', 'phantomjs-2.1.1-macosx.zip'), bin path: /Users/raf/Downloads/echo360-master/bin/phantomjs-2.1.1-macosx/bin/phantomjs
[DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection POST http://127.0.0.1:54281/wd/hub/session {"capabilities": {"alwaysMatch": {"browserName": "phantomjs", "platformName": "any"}, "firstMatch": [{}]}, "desiredCapabilities": {"platform": "ANY", "browserName": "phantomjs", "version": "", "phantomjs.page.settings.userAgent": "Mozilla/5.0 (iPad; CPU OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25", "javascriptEnabled": true}}
[DEBUG: 12-04 19:13] urllib3.connectionpool Starting new HTTP connection (1): 127.0.0.1:54281
[DEBUG: 12-04 19:13] urllib3.connectionpool http://127.0.0.1:54281 "POST /wd/hub/session HTTP/1.1" 200 699
[DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection Finished Request
[DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection POST http://127.0.0.1:54281/wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/url {"url": "https://view.streaming.sydney.edu.au:8443/ess/portal/section/2018_S2C_PHYS3069_ND", "sessionId": "7e93bb90-f79c-11e8-8675-0bc53fde1a25"}
[DEBUG: 12-04 19:13] urllib3.connectionpool Starting new HTTP connection (1): 127.0.0.1:54281
[DEBUG: 12-04 19:13] urllib3.connectionpool http://127.0.0.1:54281 "POST /wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/url HTTP/1.1" 200 74
[DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection Finished Request
[DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection POST http://127.0.0.1:54281/wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/element {"using": "xpath", "sessionId": "7e93bb90-f79c-11e8-8675-0bc53fde1a25", "value": "//*[contains(@id,'username')]"}
[DEBUG: 12-04 19:13] urllib3.connectionpool Starting new HTTP connection (1): 127.0.0.1:54281
[DEBUG: 12-04 19:13] urllib3.connectionpool http://127.0.0.1:54281 "POST /wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/element HTTP/1.1" 500 14615
[DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection Finished Request
[DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection GET http://127.0.0.1:54281/wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/source {"sessionId": "7e93bb90-f79c-11e8-8675-0bc53fde1a25"}
[DEBUG: 12-04 19:13] urllib3.connectionpool Starting new HTTP connection (1): 127.0.0.1:54281
[DEBUG: 12-04 19:13] urllib3.connectionpool http://127.0.0.1:54281 "GET /wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/source HTTP/1.1" 200 4918
[DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection Finished Request
[DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection GET http://127.0.0.1:54281/wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/source {"sessionId": "7e93bb90-f79c-11e8-8675-0bc53fde1a25"}
[DEBUG: 12-04 19:13] urllib3.connectionpool Starting new HTTP connection (1): 127.0.0.1:54281
[DEBUG: 12-04 19:13] urllib3.connectionpool http://127.0.0.1:54281 "GET /wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/source HTTP/1.1" 200 4918
[DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection Finished Request
[DEBUG: 12-04 19:13] echo360.downloader No username found (no need to login?)
[DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection GET http://127.0.0.1:54281/wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/source {"sessionId": "7e93bb90-f79c-11e8-8675-0bc53fde1a25"}
[DEBUG: 12-04 19:13] urllib3.connectionpool Starting new HTTP connection (1): 127.0.0.1:54281
[DEBUG: 12-04 19:13] urllib3.connectionpool http://127.0.0.1:54281 "GET /wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/source HTTP/1.1" 200 4918
[DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection Finished Request
[DEBUG: 12-04 19:13] echo360.downloader Dumping login page at https://view.streaming.sydney.edu.au:8443/ess/portal/section/2018_S2C_PHYS3069_ND:
<title>Echo360</title>

  <style type="text/css">

     #verticallyCentered {
        color: black;
        background-color: transparent;
        text-align: center;
        position: absolute;
        top: 50%;
        left: 0px;
        width: 100%;
        height: 1px;
        overflow: visible;
        visibility: visible;
        display: block;
     }

     #horizontallyCentered {
        font-family: Verdana, Geneva, Arial, sans-serif;
        background-color: transparent;
        margin-left: -125px;
        position: absolute;
        top: -35px;
        left: 50%;
        width: 250px;
        height: 70px;
        visibility: visible
     }

        html, body, div, iframe { margin:0; padding:0; height:100%; }
        iframe { display:block; width:100%; border:none; }
   </style>
  <link rel="shortcut icon" href="/ess/image/favicon.ico">

<script type="text/javascript" async="" src="https://ssl.google-analytics.com/ga.js"></script><script type="text/javascript">

var _gaq = _gaq || [];
_gaq.push(
['_setAccount','UA-32737747-2'],
['_setDomainName','none'],
['_setAllowLinker',true],
['_setCustomVar',1,'ESSVer','5.5.71.10.618800291'],
['_setCustomVar',2,'AH','e51e6fbe0cf29713fb4e37c5d10b46dacd90e23a24bf8fb11a5f34b2108ba74a'],
['_setCustomVar',3,'BH','634e213c0af162f0c1f067c48a1297960178627d5cde03a87ca18323daee144c'],
['_setCustomVar',4,'CH','6ffaea0b-acef-4589-9cbd-05e5fad51e13'],
['_setCustomVar',5,'DH','NULL'],
['_trackPageview']
);
_gaq.push(
['b._setAccount', 'UA-42150671-1'],
['b._setDomainName', 'none'],
['b._setAllowLinker', true],
['b._trackPageview']
);
(function() {
var ga = document.createElement('script');
ga.type = 'text/javascript';
ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0];
s.parentNode.insertBefore(ga, s);
})();
</script>

<iframe frameborder="0" src="/ess/client/section/5b42c667-4947-412e-af92-2e0bccdd0f9c?apiUrl=https%3A%2F%2Fadmin.streaming.sydney.edu.au%3A8443%2Fess&userID=anonymousUser&token=39f4f9968844d9cd4a0d4d46c7e4012febd92c00e3edb97b3694c578cfbeebcb21605c1bc993b09ff66326bd4e0c6212733c6889ac0f0002b5453548a872a078bcd6a7a54c84862ca27f6f87cb0ca205f13be3f22dbe9692223b99c9bc4388d2b0a2c22458a83b55f5205c0dec1d57dc182a550934e388386f98541466040e9f640afaee9ad5c3e0&contentBaseUri=https%3A%2F%2Fview.streaming.sydney.edu.au%3A8443%2Fess" scrolling="auto"></iframe>

<script id="f5_cspm">(function(){var f5_cspm={f5_p:'IHEGECCJNFFNJFDLJBGPKLLFBBHEEJPAHPAJKIEJEELBBODGMIDBPHDDOGBAGOCIAAKJELFMBIBCNPOFAAHJBBPFFEENDBNANOONIDGJHMEIKHGLKABDIFHFLHIBGKHH',setCharAt:function(str,index,chr){if(index>str.length-1)return str;return str.substr(0,index)+chr+str.substr(index+1);},get_byte:function(str,i){var s=(i/16)|0;i=(i&15);s=s*32;return((str.charCodeAt(i+16+s)-65)<<4)|(str.charCodeAt(i+s)-65);},set_byte:function(str,i,b){var s=(i/16)|0;i=(i&15);s=s*32;str=f5_cspm.setCharAt(str,(i+16+s),String.fromCharCode((b>>4)+65));str=f5_cspm.setCharAt(str,(i+s),String.fromCharCode((b&15)+65));return str;},set_latency:function(str,latency){latency=latency&0xffff;str=f5_cspm.set_byte(str,32,(latency>>8));str=f5_cspm.set_byte(str,33,(latency&0xff));str=f5_cspm.set_byte(str,27,2);return str;},wait_perf_data:function(){try{var wp=window.performance.timing;if(wp.loadEventEnd>0){var res=wp.loadEventEnd-wp.navigationStart;if(res<60001){var cookie_val=f5_cspm.set_latency(f5_cspm.f5_p,res);window.document.cookie='f5avr1986603928aaaaaaaaaaaaaaaa='+encodeURIComponent(cookie_val)+';path=/';} return;}} catch(err){return;} setTimeout(f5_cspm.wait_perf_data,100);return;},go:function(){var chunk=window.document.cookie.split(/\s*;\s*/);for(var i=0;i

[DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection POST http://127.0.0.1:54281/wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/url {"url": "https://view.streaming.sydney.edu.au:8443/ess/client/api/sections/2018_S2C_PHYS3069_ND/section-data.json?pageSize=100", "sessionId": "7e93bb90-f79c-11e8-8675-0bc53fde1a25"}
[DEBUG: 12-04 19:13] urllib3.connectionpool Starting new HTTP connection (1): 127.0.0.1:54281
[DEBUG: 12-04 19:13] urllib3.connectionpool http://127.0.0.1:54281 "POST /wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/url HTTP/1.1" 200 74
[DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection Finished Request
[DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection GET http://127.0.0.1:54281/wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/source {"sessionId": "7e93bb90-f79c-11e8-8675-0bc53fde1a25"}
[DEBUG: 12-04 19:13] urllib3.connectionpool Starting new HTTP connection (1): 127.0.0.1:54281
[DEBUG: 12-04 19:13] urllib3.connectionpool http://127.0.0.1:54281 "GET /wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/source HTTP/1.1" 200 238
[DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection Finished Request
[DEBUG: 12-04 19:13] echo360.course Dumping course page at https://view.streaming.sydney.edu.au:8443/ess/client/api/sections/2018_S2C_PHYS3069_ND/section-data.json?pageSize=100:

{"errorMessage":"Invalid user session.","error":401}

[DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection POST http://127.0.0.1:54281/wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/element {"using": "tag name", "sessionId": "7e93bb90-f79c-11e8-8675-0bc53fde1a25", "value": "pre"}
[DEBUG: 12-04 19:13] urllib3.connectionpool Starting new HTTP connection (1): 127.0.0.1:54281
[DEBUG: 12-04 19:13] urllib3.connectionpool http://127.0.0.1:54281 "POST /wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/element HTTP/1.1" 200 104
[DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection Finished Request
[DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection GET http://127.0.0.1:54281/wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/element/:wdc:1543911215300/text {"sessionId": "7e93bb90-f79c-11e8-8675-0bc53fde1a25", "id": ":wdc:1543911215300"}
[DEBUG: 12-04 19:13] urllib3.connectionpool Starting new HTTP connection (1): 127.0.0.1:54281
[DEBUG: 12-04 19:13] urllib3.connectionpool http://127.0.0.1:54281 "GET /wd/hub/session/7e93bb90-f79c-11e8-8675-0bc53fde1a25/element/:wdc:1543911215300/text HTTP/1.1" 200 132
[DEBUG: 12-04 19:13] selenium.webdriver.remote.remote_connection Finished Request

[soraxas]

Hi @rafiulnakib the updated script should works now, please delete the entire old folder, redownload it from Github, and try again :) please star this if it helps you too. Cheers

[rafiulnakib]

Worked perfectly! Thanks.