Add redacted livestream archive files

[benhylau]

Daily livestream archives received from @darkdrgn2k. These already have some talks redacted as needed.

@garrying I would like these files to be hosted on https://2019.ournetworks.ca/livestream/ in our 2019 archived site. Where should I put them?

@darkdrgn2k my understanding is we don't need to alter the gateway here. We can simply set the env var in the player to use infura and it will replace all the URLs as they play?

Resolves #157

[dcwalk]

Can we instead have the stream listed on the recorded talks page? I was anticipating pulling down the stream pages as we archive the 2019 version of the site

…

On Thu, Sep 26, 2019 at 3:55 PM Benedict Lau ***@***.***> wrote: Daily livestream archives received from @darkdrgn2k <https://github.com/darkdrgn2k>. These already have some talks redacted as needed. @garrying <https://github.com/garrying> I would like these files to be hosted on https://2019.ournetworks.ca/livestream/ in our 2019 archived site. Where should I put them? ------------------------------ You can view, comment on, or merge this pull request online at: #154 Commit Summary - Add redacted livestream archive files File Changes - *A* livestream/live-2019-09-20.m3u8 <https://github.com/ournetworks/ournetworks.ca/pull/154/files#diff-0> (529) - *A* livestream/live-2019-09-21.m3u8 <https://github.com/ournetworks/ournetworks.ca/pull/154/files#diff-1> (1268) - *A* livestream/live-2019-09-22.m3u8 <https://github.com/ournetworks/ournetworks.ca/pull/154/files#diff-2> (1133) Patch Links: - https://github.com/ournetworks/ournetworks.ca/pull/154.patch - https://github.com/ournetworks/ournetworks.ca/pull/154.diff — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#154?email_source=notifications&email_token=ABDZWYXUQEOYPTPNPEPFBCDQLUHTHA5CNFSM4I26MXL2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HN7N6WA>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABDZWYQARR7VWQJG4LJHL43QLUHTHANCNFSM4I26MXLQ> .

[benhylau]

Links that are tweeted out via share buttons are of the format:

https://2019.ournetworks.ca/livestream/?m3u8=live-2019-09-20.m3u8&from=QmU1Wd6uJdB35uWh9upnU5DF5PeZDekSkjtTEHUpXSYZY5

e.g. https://twitter.com/LauBenedict/status/1175163389032718336

This means the player will look for live-2019-09-20.m3u8 at https://2019.ournetworks.ca/livestream/ so while we can list the archive link on the recorded talks page, the player should play from https://2019.ournetworks.ca/livestream/?m3u8=live-2019-09-20.m3u8

Of course we can hack up the player to not respect the m3u8 parameter, but I would rather not do that.

[garrying]

I think these m3u8 files should go somewhere else so that /livestream is just the web player instead of player + playlists. Could we put them in a folder named something like /playlists? or /media?

[benhylau]

I think playlist at player root was the intention. The links are already tweeted out with m3u8=live-2019-09-20.m3u8 not m3u8=playlist/live-2019-09-20.m3u8, so doing this another way would require some weird prepending of playlist/ in front of the param in the player, which I rather not do.

In the past, we have put the video at player root:

# ipfs ls QmWsKvBvXUKaHcHzrUS91XV4k3YjQFdywQ7bY9BZVX4ghk
QmVnVomSomrHoV3QpCHyb6T2m8ex6ZqwmVezJyDXiYEci8 984       index.html
QmZBD7HdWkdBCB7fk7QX6wVqfpPpSSa11TzKD3SBykLmqw 36570     poster.png
QmXSz5L3d1zziP3UVTVBzPaGvkHJzVAE68V9kQVaN7ZrgG 1150      reset.css
QmdtMX4jKsfX5JoagJhxnrbBXUUERxE2fgnKoicy7vYQkz 27138     video-js.css
QmT72tXZYBxzdccyuabZnCCbbpZyoftb5DLegZqmYUfMqB 203869094 video.mp4
QmcGYGxbvncGK2ftcM764M14AF4pEfEWpC4GBBYtAFWTG3 68824     video.plugin.js
QmXMaSaanyLF7UQ5b74p18j1LZDkTrMCyJQKFJsAHSG1zU 4815      vjs.eot
QmVprfEknr45ke7Tr29j9mvAag9sEm3fKbGeJY2cpLuCUa 10367     vjs.svg
QmQVt3fftGoGfZPx9LQoKTkyjXaWLyCrhX8RQFe8XtqrHg 4651      vjs.ttf
QmdZJjKF8Acimsjdd16cnHnF3GuMG4Qza6aPfw45685B28 3503      vjs.woff

This was the ipfs show and tell so I feel keeping all these at livestream root would be appropriate?

[benhylau]

@darkdrgn2k I think we want to take these from ipfs.infura.io once we have them pinned right? If we are going to stream from here, we can probably keep them as relative?

[darkdrgn2k]

new code seems to correct the issue that would arise from this

Tested using sed "s|https://ipfs-gateway.live.mesh.world||" test1.m3u8 and it seemed to work fine

However it re-append an ipfs server unless you clear out the value in
ipfs_gateway

[benhylau]

How do you think I should get these onto ipfs.infura.io / ipfs.io? Now there are 1449 hashes and I am doing:

$ grep 'ipfs/' live-*.m3u8 | sed 's|.*ipfs/||g' > ipfs-cids.txt
$ cat ipfs-cids.txt | xargs -I {} curl "https://ipfs.infura.io:5001/api/v0/pin/add?arg=/ipfs/{}&recursive=true&progress=true" >> ipfs-cids.log

Which is taking forever and I don't think infura is actually able to locate some of the chunks... they are all pinned on live.mesh.world right?

[benhylau]

@darkdrgn2k the ipfs_gateway param is a little different in the Our Networks fork, so this is working now :D

[benhylau]

Live Stream Archive and bookmarks now working with:

• m3u8 (e.g. live-2019-09-20.m3u8) coming from static local directory livestream/
• ts chunks coming from ipfs.io gateway by default
• ts chunks gateway can be overridden with gw param (e.g. &gw=https://ipfs.infura.io)

You can try with this PR:

http://localhost:4000/livestream/?m3u8=live-2019-09-20.m3u8&from=QmU1Wd6uJdB35uWh9upnU5DF5PeZDekSkjtTEHUpXSYZY5

Notice that share link is also fixed to show the m3u8 file passed through URL param.

All chunks are uploaded/uploading to ipfs.io and ipfs.infura.io via ournetworks/2019#128 (comment) and we are completely independent from live.mesh.world now with this PR.