Skip to content
This repository has been archived by the owner on Mar 25, 2020. It is now read-only.

Post-process videos #128

Closed
3 tasks done
dcwalk opened this issue Sep 25, 2019 · 30 comments
Closed
3 tasks done

Post-process videos #128

dcwalk opened this issue Sep 25, 2019 · 30 comments
Labels
post-event stream

Comments

@dcwalk
Copy link
Member

dcwalk commented Sep 25, 2019 •
edited
Loading

A tracking issue for the work of processing the videos. Pulling on previous notes from August:

  • Produced video time estimation for post-processing
    • Two streams of video + 1 stream of audio
    • Estimate 2 hours per talk but depends on talk length
    • Need to watch them at least once
    • 22 talks, 15.5 hours of recordings
      • 2 orga
      • 2 lightning
      • 18 presenters
    • ~30 min overhead to set up each video:
      • Lower thirds
      • Metadata
      • Project setup (using templates this year)
    • Post-processing & publishing plan
      • 0.5 hour * 22 talks + 15.5 hour * 1.5 padding < 40 hour
      • Decouple post-processing work from publishing work
      • hank: to handle post-processing
      • dc: handle publishing plan

So, we are talking ~40 hours, with the expectation that @dcwalk will take care of publishing planning (separate issue)

Timeline

Begin: September 30
End:

Todos
@Shrinks99
Copy link

Here's the link to view the tracking spreadsheet: https://docs.google.com/spreadsheets/d/1ovifZ1tLCy3m6fJGmabhOkpZ385ICk5EdrKDHcOD0Yw/edit?usp=sharing

I've finished all the templating stuff, will look at my schedule later today and fill out the due dates accordingly. Also all the text in the intro (sans logo) uses the cool build-in typewriter effect that the conference title uses. Neat!

@dcwalk
Copy link
Member Author

dcwalk commented Sep 30, 2019

Thanks for sharing your tracking sheet.

I've finished all the templating stuff, will look at my schedule later today and fill out the due dates accordingly. Also all the text in the intro (sans logo) uses the cool build-in typewriter effect that the conference title uses. Neat!

I'm not sure what this means, I thought you and @garrying had locked down the splash/intro stuff, could you post if you are changing?

@Shrinks99
Copy link

Shrinks99 commented Oct 1, 2019 •
edited
Loading

The splash / intro is nearly identical, the talk title and name just build in the same way as the conference title does instead of fading in. The templating bit is me making our designs work in Premiere so I can just copy paste text into a box and not have to re-render everything every time.
https://youtu.be/ujDbH52LVq8

@Shrinks99
Copy link

Day 1 talks are all finished! I plan to edit the rest of these on reading week which starts on the 14th but I'll be back at my workstation on the 16th (guaranteed) so that's when I plan to start the bulk of the work. The spreadsheet has been updated with the dates.

Should I begin uploading what I have to YouTube? Do we want to schedule when these start to release? I'd really like to not upload these all in one go and release them once a day as to not spam everyone's YouTube feeds like The Linux Foundation just did because that's a bummer and nobody likes that :P
Screenshot_2019-10-03_20-33-38

@Shrinks99
Copy link

@dcwalk For Dr. Jennifer Wemigwans' talk would you like the territorial acknowledgement left in the video or would you like me to start the talk when she begins speaking?
Once I know this I would like to publish it to YouTube as an unlisted video so I can send it to her for approval so that it is approved (or not, we'll see what she says) in time for release. She was the only presenter that wanted to review it first.

@dcwalk
Copy link
Member Author

dcwalk commented Oct 18, 2019

If you think it flows to leave the land acknowledgement/intro in I think that would be preferrable.

Your plan sounds good, maybe you can propose your approach to her by email before publishing as unlisted and sharing? I'm not sure about the degree of concern/sensitivity here

@Shrinks99
Copy link

It works fine with it in, just wanted to make sure you were on-board with being included in the video since it's a little more than the usual introduction or ending bits. It flows well!

I think Jennifer just wanted to review the video before we released it to make sure she didn't do a bad job speaking. Unlisted videos can only be seen to those who have the link and we just won't share the link with anyone else (same thing as all my intro previews which can't be seen on my channel). If she likes it we will schedule the release to become public like all the other videos and if she doesn't it gets deleted off the channel. I'll make sure this is noted in my email to her so it's clear that the YouTube link we're sending isn't publicly accessible.

That aside, all talks should be finished today! I can start uploading them and setting up the descriptions and tags on YouTube as well if you'd like, they'll all be private until a release schedule is decided upon.

@dcwalk
Copy link
Member Author

dcwalk commented Oct 18, 2019

Unlisted videos can only be seen to those who have the link and we just won't share the link with anyone else (same thing as all my intro previews which can't be seen on my channel).

Yes I understand that, my comment about waiting before uploading it is trying to be sensitive to the perception of checking in before moving forward.

@Shrinks99
Copy link

Would you like me to share the video with Jennifer using another method? I can probably give her a Dropbox or Google Drive link if you think that's better than YouTube for this.

@dcwalk
Copy link
Member Author

dcwalk commented Oct 18, 2019

This could be a question you ask her. I don't have a preference.

@Shrinks99
Copy link

Editing is finished for all talks without blockers! Jennifer's talk is finished editing and I've emailed her (and CCed the org email), the one talk with a blocker is Zach's (Zack's?) lightning talk because I don't definitively know how to spell his first name and unfortunately he didn't have his conference lanyard on during the talk. Once I get some sort of answer there I can plop in the titles and render.

Other than that, thumbnails need to be generated and descriptions need to be put into text files and then we're done! MUCH nicer process compared to last year and overall I'm quite happy :D

@benhylau
Copy link
Member

I think @Shrinks99 has valid points about scheduled releases on YouTube. One the other hand, we are almost ready to make a big announcement to have everything ready on every platform, including our 600 Twitter + 300 mail subscribers, compared to our < 100 on YouTube. That said, scheduled releases could let YouTube better promote our content to attract more subscribers.

My preference is to release everything on IPFS, Internet Archive, and our partner ISOC, but schedule YouTube releases. In the newsletter announcement, tell people everything is on those platforms, and YouTube releases are scheduled over next couple weeks, and those releases have in their description link backs to our non-Google archives. This way we are fully utilizing YouTube's network effect, while giving a bias to the platforms we prefer.

@benhylau
Copy link
Member

Next steps:

  1. Get all post-processed assets from @Shrinks99
  2. Prepare files for organizers to review and track review status on spreadsheet
  3. Release to Infura IPFS, Internet Archive, and YouTube* by Nov 15
  4. Hand-off to @garrying for website linking** and organizers to draft announcement email

*By Nov 15, we need YouTube links of all talks available on website as in 2018, but it seems we cannot keep them as unlisted and then schedule for them to become publicly listed. If that's the case we'd have to release entire archive together on Nov 15. If there is a good way around this then we can schedule to release on YouTube one per day.

**@garry and I will also sort out how to link the livestream archive m3u8 such that bookmark links from before are preserved, and @darkdrgn2k and I probably need to move chunks to a more permanent place than our server

@Shrinks99
Copy link

Shrinks99 commented Oct 29, 2019 •
edited
Loading

*By Nov 15, we need YouTube links of all talks available on website as in 2018, but it seems we cannot keep them as unlisted and then schedule for them to become publicly listed. If that's the case we'd have to release entire archive together on Nov 15. If there is a good way around this then we can schedule to release on YouTube one per day.

Huh, so this works a little differently than I thought it would. If you set the video to Private you can schedule an upload time but if it's just Unlisted you cannot. If you manually set a video from Unlisted to Public it will go out to subscribers as a regular video upload would.

Alternatively, the YouTube links could just be put up on the website in one batch after they are all public if we don't want to manually make one video public every day which is a pretty dumb way of doing it but that would also work.

As for the spreadsheet, I can give you all access to it and you can mark down in the progress column what the status is?

@benhylau
Copy link
Member

I am definitely not going to manually unlist one video per day :D and we have set a hard deadline to get the 2019 site archived under the 2019.ournetworks.ca so it seems not worth it to do scheduled listing. We may as well publish the entire archive on Nov 15 so we have all the links for website.

Yes please give me access to spreadsheet!

@Shrinks99
Copy link

Videos have been uploaded to the Alexandria server with the same naming conventions as the 2018 videos. Descriptions will be uploaded (taken from the website or created by me for the lightning talks) soon!

@benhylau
Copy link
Member

benhylau commented Nov 6, 2019 •
edited
Loading

Hank on chat:

benhylau: you’ve been added to the sheet, pls use the same column with the colours in it and set to Approved once the video is good, if it’s not good fill with yellow and set it to Blocked with a reason why in the notes

Thanks @Shrinks99, I will follow the below checklist to review the archive, and add notes to a new column I created called Reviewer Notes so I don't mix up your production notes. Here is the spreadsheet link.

For each video:

  • 1. Verify the filenames have the correct spellings, consistent casings, and match between the 1080p, 720p, and descriptions folders
  • 2. Verify the description is in a text file consistent with the 2018 format and contains the text from GitHub submissions unless it was revised on the website
  • 3. Verify the session and presenter names in video are as listed on program
  • 4. Verify the video beginning and end are correct
  • 5. Verify the video and audio sync at different points of the video

For the archive:

  • 6. Verify all videos we expect to publish are included
  • 7. Verify we are not including any video that should not be published

When all these verifications are completed, we'll be publishing the entire archive to:

  • Alexandria, already there but need to IPFS pin (e.g. 2018)
  • IPFS, on Infura.io (e.g. 2018)
  • Internet Archive (e.g. 2018)
  • YouTube (e.g. 2018)

Updates:

  • Fixed all issues I found with filenames, see notes in spreadsheet
    • For Channel Channel, the a network for VPN access text was missing, I left it as is, not sure if we want to change it or not
    • LoRaWAN vs. LoRAWAN in video rendering
  • Descriptions folder was empty, so I went through GitHub and program to compile all of them, would like someone else to verify
    • I included at the bottom both project and personal links that are associated with the session
    • For sessions tied to art exhibits, I added a line at the top to mention the association
    • Replaced all characters in text that don't render correctly, except é in Renée's name I left as is
    • Two links are dead, I left them in since they were provided by presenters
    • Farhad's talk matches the website, not GitHub, due to changes not reflected in GitHub since submission
    • Descriptions were mostly taken from GitHub otherwise
    • I wrote the lightning talk descriptions, which basically says it's a lightning talk, except the mesh ones has a bit more detail
    • The keynotes, I took from the website, and added panelists in the May First, but did not indicate they are keynotes

@dcwalk
Copy link
Member Author

dcwalk commented Nov 12, 2019

@benhylau -- we should work from the data file for sessions for all descriptions, presenter names, etc..., as it builds the recorded talks page.

Can the txt files be updated to mirror what is here: https://github.com/ournetworks/ournetworks.ca/blob/master/_data/sessions.yml?

As you are the one doing that work I'm not sure what I'm supposed to verify. I've reviewed the production spreadsheet prior and confirmed that no talks were included that shouldn't be. It doesn't seem like I need to review again

@benhylau
Copy link
Member

The two mostly match, but not 100% as they are intended to be pasted into YouTube descriptions, etc. with project links and context information. For example...

Our_Networks_2019_Channel_Channel_by_Biyi_Wen.txt contains

Video presentation by Biyi Wen, accompanying the exhibited artwork: Channel channel, a network for VPN access.

which I added.

Our_Networks_2019_DHT_BFT_OMG_Know_Your_Building_Blocks_For_Decentralized_Applications_2019_Edition_by_Kirill_Pimenov.txt contains

It is going to be a revised and extended variant of my talk on Web3 Summit in Berlin in 2018: https://www.youtube.com/watch?v=J2lnjFqi-No

which is in the original GitHub submission. Other submissions have links that I extracted to put into the description, some giving important context such as in Our_Networks_2019_Building_A_Modern_Memex_by_Andrew_Louis.txt.

Our_Networks_2019_The_Impotent_Medium_by_Farhad_Bahram.txt has more information about the video, whereas in sessions.yml it just says

Video presentation.

Then there are items like lightning talks, where it isn't a 1:1 mapping, as talks are broken up into individual video files, whereas on sessions.yml it is a single record.

@dcwalk is it important that they match 100%? I'd prefer descriptions in the archive to provide best context of the video on their own.

Matching sessions.yml means you'd like the Keynote text to be included in the filenames and video title page right? cc. @Shrinks99 this probably means we need to re-render both keynotes and Channel Channel?

title: "Keynote: Dr. Jennifer Wemigwans, A Digital Bundle Protecting and Promoting Indigenous Knowledge Online"
title: "Keynote panel: May First, technology and revolution"

Regarding 6 and 7, I have checked that but want a second person to verify. If you feel the production spreadsheet is correct then we're fine, as I have verified that it matches the archive.

@dcwalk
Copy link
Member Author

dcwalk commented Nov 12, 2019 via email

@dcwalk
Copy link
Member Author

dcwalk commented Nov 12, 2019 via email

@Shrinks99
Copy link

Shrinks99 commented Nov 13, 2019 •
edited
Loading

I changed the keynote title name because it didn't follow the format used in the other videos. I would propose that we follow the format used for everything else to keep things consistent. For the panel I felt that it's important to signify that it's a panel in the title and I suppose the filename should be updated to match. This is sort of subjective and up for debate though so I'll defer to what you want on both cases. I would highly suggest that we don't put "Dr. Jennifer Wemigwans" in the video title however or else we'll be left with two instances of it in the title and that seems foolish. In any case, it seems that we are in agreement :P

@benhylau
Copy link
Member

I have reviewed and approved all videos after @Shrinks99 re-published the incomplete files. I think we're good to go and have updated #128 (comment) to reflect that 🎉

@Shrinks99 can you start the upload to YouTube, using descriptions from here. I will wait for a final decision re: Jennifer's talk then publish to IPFS pin on Alexandria and Infura.io, and publish to IA.

@benhylau
Copy link
Member

Fixed mismatched filenames and confirmed they now match across directories. Useful commands for the verification:

# ls descriptions | sed 's/.txt//g' | sha1sum
32cbf94c27da6c77986475c35994fd3097a281bd  -
# ls 720p | sed 's/_720p.mp4//g' | sha1sum
32cbf94c27da6c77986475c35994fd3097a281bd  -
# ls 1080p | sed 's/_1080p.mp4//g' | sha1sum
32cbf94c27da6c77986475c35994fd3097a281bd  -

Changed file owner and permissions to be same as 2018 on alexandria.

The archive to be published has the following sha1sum:

# find ./descriptions -type f -print0 | sort -z | xargs -0 sha1sum | sha1sum
ef3ccb9bf4825c3a921336b974dac8df5bb5c9ff  -
# find ./720p -type f -print0 | sort -z | xargs -0 sha1sum | sha1sum
a5d7e17b4b3dd5e70105a36a5734c3b2ecd5ca2b  -
# find ./1080p -type f -print0 | sort -z | xargs -0 sha1sum | sha1sum
e5db760fbcd19b05a887fb17774106addc8c1b94  -

@Shrinks99 can you confirm these are the same on your local? This is excluding Jennifer's talk in all folders.

@Shrinks99
Copy link

I will be able to confirm these changes and start uploading on November 23rd.

@Shrinks99
Copy link

All of the talks have been uploaded to YouTube! They are currently all up as unlisted on our channel.

YouTube has changed their system and it seems that we can now schedule videos to publish while they remain unlisted, this would allow us to release one video per day on YouTube while keeping the video URLs which can be accessed through the website or the conference playlist.

I'll get on thumbnails ASAP and hopefully we can push this thing out the door soon!

@benhylau
Copy link
Member

benhylau commented Dec 2, 2019 •
edited
Loading

IPFS pinning recorded videos

After @Shrinks99 used sftp to transfer all files to Alexandria, and @benhylau did file name and content verifications, we started using Alexandria as IPFS origin, with the following procedures.

alexandria.tomesh.net

Used following command in screen to pin 2019 archive on Alexandria. Note --local deprecation shouldn't affect results:

benhylau@alexandria:~$ sudo su ipfs
ipfs@alexandria:/home/benhylau$ cd /mnt/sda1/data/ournetworks/2019
ipfs@alexandria:~/data/ournetworks/2019$ ipfs add --local=true --nocopy --fscache -r --progress .
09:22:26.939 ERROR core/comma: Command 'add', --local is deprecated, use --offline instead env.go:40
...
added QmaWc83FvLkyvw8p7Q45prW7HTXAaLgk8jxmkdciokaE1K 2019
 62.68 GiB / 62.68 GiB [=============================================================================================================================] 100.00%

File hashes:

added QmWj2qwrGAG43F6zgAYhQGkXHBQpfLWfMGhMpEuLhcVxCB 2019/1080p/Our_Networks_2019_A_Digital_Bundle_Protecting_And_Promoting_Indigenous_Knowledge_Online_by_Dr_Jennifer_Wemigwans_1080p.mp4
added QmVEW7WjJsh8W9EGYfq5d4kjJ2XzDDL5ZQy6VXZ2LcMXZt 2019/1080p/Our_Networks_2019_A_Technical_Introduction_To_IPFS_by_Hector_Sanjuan_1080p.mp4
added QmNzJxxxwbKADxW4uhqMiaRMCDkUjDTPkFr9kii9fpb6xF 2019/1080p/Our_Networks_2019_Building_A_Modern_Memex_by_Andrew_Louis_1080p.mp4
added QmSRBcvpYdK6tiLjY7DVepj6VgFZZDvB2uJK7CUyjh2B2x 2019/1080p/Our_Networks_2019_Channel_Channel_by_Biyi_Wen_1080p.mp4
added QmRJcehUBNZnDUvSHJg97Aeus8AXw5rtF1oxMtRYkevBwV 2019/1080p/Our_Networks_2019_Collective_Resistance_To_State_Controls_On_Information_by_Cecylia_Bocovich_1080p.mp4
added QmYNSchxz3GEZdmcSVFWipp3qdKxUfQETck3ANE1CqYLBm 2019/1080p/Our_Networks_2019_Cryptoeconomics_Towards_Mass_Behavioral_Engineering_Or_A_Network_Commons_by_Jaya_Klara_Brekke_and_Dan_Hassan_1080p.mp4
added QmPYoWZA9xGhhrGjnZZnoa5FLWoNq6LVo4xmS8reQSqu4Q 2019/1080p/Our_Networks_2019_DHT_BFT_OMG_Know_Your_Building_Blocks_For_Decentralized_Applications_2019_Edition_by_Kirill_Pimenov_1080p.mp4
added Qmf7WSGo1ftiKy8fwQTEDfS984zbW8VZCE8Jkgd76gn4gQ 2019/1080p/Our_Networks_2019_Distributed_Internet_Archiving_Plus_Low_Power_Long_Distance_Meshing_With_LoRaWAN_by_Nick_Sweeting_1080p.mp4
added Qmbz4a5JLjEwAb2ZhdaJCVKKpFWYt1yWKooqpb82amjrAR 2019/1080p/Our_Networks_2019_Distributed_Secure_Routing_In_Permissionless_Flat_Networks_by_Benedict_Lau_1080p.mp4
added QmWP6pQKg8GhkCF2e9SSZP6H8tXZ9vUh6LWG6H4paauDoP 2019/1080p/Our_Networks_2019_Distributing_Knowledge_Of_Digital_Workflows_Via_Workshops_And_Support_Groups_by_Zack_1080p.mp4
added QmXkxhpzNEYaLXS3upFmutdrELLAm9WdFmxYTxNYXwwbhN 2019/1080p/Our_Networks_2019_Hugo_Wasm_Plus_IPFS_Blogging_by_Jim_Pick_1080p.mp4
added QmUUbsQgYpEhVMFdww32zLSKyu5D8zQPn5MLx8TZ6oXVQm 2019/1080p/Our_Networks_2019_May_First_Technology_And_Revolution_by_Alfredo_Lopez_and_Ellie_Ade_Kur_and_Lorraine_Chuen_and_Victoria_Barnett_1080p.mp4
added QmVM6zFukpU1jMdmQjwPuBvqYE4DHurti9yofGaJLXX8kg 2019/1080p/Our_Networks_2019_Mesh_Lightning_Talks_Freifunk_by_Felix_Kaechele_1080p.mp4
added QmUFmpwLBLsPhT1XbCrC2PySZ6YHCxFz7XetLVd6ekNfEM 2019/1080p/Our_Networks_2019_Mesh_Lightning_Talks_I2P_Intro_For_App_Devs_by_Idk_1080p.mp4
added QmSYWZLRuy14iVSyEg1qJ2s5K16qxQhc4AJqL3HF3ekvbr 2019/1080p/Our_Networks_2019_Mesh_Lightning_Talks_NYC_Mesh_by_Jillian_Murphy_and_Scott_Rasmussen_1080p.mp4
added QmVzAoss8soUmxGUFnB9Gp95cQb2itdwHDFdVFVx6x97k4 2019/1080p/Our_Networks_2019_Mesh_Lightning_Talks_Sarantaporogr_by_Vassilis_Chryssos_1080p.mp4
added QmctVZnwgnZSjhNBiZbW4nrfLAqBM9eu3dPhCUik6FXVqt 2019/1080p/Our_Networks_2019_Networks_As_Sculpture_by_Lai_Yi_Ohlsen_1080p.mp4
added QmRmD44SiKNFTJqsZbzLneFQPuTRv2iZy9n9MFZRLQYcG1 2019/1080p/Our_Networks_2019_Regulationism_And_The_Need_For_A_Digital_Postmodernism_by_Corey_Abell_1080p.mp4
added QmcBJpcFs7DA8Uuz9MG9TzCt1FrdAKoadNeY7LcVPa6wDn 2019/1080p/Our_Networks_2019_Remaking_The_Web_Competing_Visions_Of_Decentralization_by_Jack_Jamieson_1080p.mp4
added QmetkRpcPihomCTfdLZcdmj3sGMvea1FawjR7neipVYhQt 2019/1080p/Our_Networks_2019_Social_Artistic_And_Theoretical_Experiments_With_Decentralized_Festivals_by_Andreas_Dzialocha_1080p.mp4
added QmYtsm3p4JRRwbsafcUdYAm9iMf3P8swDkyepAAVuR4R6P 2019/1080p/Our_Networks_2019_Solar_Powered_Website_by_Marie_Otsuka_and_Lauren_Traugott-Campbell_1080p.mp4
added QmUJs828esqruJkh1d1PNM3XAgWdsKZe2a22nB1Eo8NpBy 2019/1080p/Our_Networks_2019_Spectacular_Grammar_Infrastructure_As_A_Universal_Language_by_Renee_Reizman_1080p.mp4
added QmYcPssQWEGxKavRAgSZ1sCgZCxE6tsTASdetyTLyWjpZb 2019/1080p/Our_Networks_2019_The_Impotent_Medium_by_Farhad_Bahram_1080p.mp4
added QmcR7xWG3unc4Jhd4MefpWwnLCjkRnMgQ4tCdzdjt3ebND 2019/1080p/Our_Networks_2019_The_Netifesto_by_Ryan_Kelln_1080p.mp4
added QmdsegrRgaMMaosnVGG4LxABPsyaS1FWgTwsVX5SMmxx4u 2019/720p/Our_Networks_2019_A_Digital_Bundle_Protecting_And_Promoting_Indigenous_Knowledge_Online_by_Dr_Jennifer_Wemigwans_720p.mp4
added QmNtQiSK7dZbMCH89kgBLzgfMQP3UoWqT2QKWvjtVt2jTx 2019/720p/Our_Networks_2019_A_Technical_Introduction_To_IPFS_by_Hector_Sanjuan_720p.mp4
added QmVCe8TMff7DJGhP9ng9bCu8mMMC2TzGWeXjAaWhATc8A1 2019/720p/Our_Networks_2019_Building_A_Modern_Memex_by_Andrew_Louis_720p.mp4
added QmTWyWDACanmeZvXnaMQPepeYbV6jkjh1eDdkYSnZFKX4A 2019/720p/Our_Networks_2019_Channel_Channel_by_Biyi_Wen_720p.mp4
added QmUNXRGpieHHNukZiNPEDP38V9tP4UeVZX6cn8Rv4spLgh 2019/720p/Our_Networks_2019_Collective_Resistance_To_State_Controls_On_Information_by_Cecylia_Bocovich_720p.mp4
added QmbaZt9qHGWdvDKsw6bkonwheE7uXbGwxd9B7UW8FBzW4Y 2019/720p/Our_Networks_2019_Cryptoeconomics_Towards_Mass_Behavioral_Engineering_Or_A_Network_Commons_by_Jaya_Klara_Brekke_and_Dan_Hassan_720p.mp4
added QmdZG4nE1v1Fiwh7ixcxcDynSZ7yCr2K52oozzHeEvpaTy 2019/720p/Our_Networks_2019_DHT_BFT_OMG_Know_Your_Building_Blocks_For_Decentralized_Applications_2019_Edition_by_Kirill_Pimenov_720p.mp4
added QmWCKia9VApWRG2FfWKZ8d2ca8c7MoSLo9VauWcrK6EWf6 2019/720p/Our_Networks_2019_Distributed_Internet_Archiving_Plus_Low_Power_Long_Distance_Meshing_With_LoRaWAN_by_Nick_Sweeting_720p.mp4
added QmawHWf4km3ZLE59S6X5FBFhZdCuqmYQ5JN7fz2d2P2ZMM 2019/720p/Our_Networks_2019_Distributed_Secure_Routing_In_Permissionless_Flat_Networks_by_Benedict_Lau_720p.mp4
added QmR7it52jCvLxjnzwyr6ADncDZuiRzZZywzBKhmfuRxYfN 2019/720p/Our_Networks_2019_Distributing_Knowledge_Of_Digital_Workflows_Via_Workshops_And_Support_Groups_by_Zack_720p.mp4
added QmTKg5WLLpW8PPSMSs5TWv9gcC1t2nrtd4drus3V2Fmxcq 2019/720p/Our_Networks_2019_Hugo_Wasm_Plus_IPFS_Blogging_by_Jim_Pick_720p.mp4
added QmX73g5nqUM2NxEkTA82F2aLLM6DPe9PMMB3kF7W825eER 2019/720p/Our_Networks_2019_May_First_Technology_And_Revolution_by_Alfredo_Lopez_and_Ellie_Ade_Kur_and_Lorraine_Chuen_and_Victoria_Barnett_720p.mp4
added QmX4yR7WiZMNGBpXAZWXS4C9gPgtrftdMpzJePB71yvFJY 2019/720p/Our_Networks_2019_Mesh_Lightning_Talks_Freifunk_by_Felix_Kaechele_720p.mp4
added QmcVUREbLmBGNhNi9zqbc6QnCDWZpGwfqdwSSU8TLwpPSD 2019/720p/Our_Networks_2019_Mesh_Lightning_Talks_I2P_Intro_For_App_Devs_by_Idk_720p.mp4
added QmTbFyRC8MvBEVk4n8CSw2G5ntvsDjG8tuzgjQAbBtCUmj 2019/720p/Our_Networks_2019_Mesh_Lightning_Talks_NYC_Mesh_by_Jillian_Murphy_and_Scott_Rasmussen_720p.mp4
added QmZAeeRadrg1x9NXQzCW17xKR3rfF3ZNuAW4rFaQrSmQFu 2019/720p/Our_Networks_2019_Mesh_Lightning_Talks_Sarantaporogr_by_Vassilis_Chryssos_720p.mp4
added QmPcqEp6RLxAHQtGDPPqziTKQvFZGfUSGxQ6mBAaSG5atd 2019/720p/Our_Networks_2019_Networks_As_Sculpture_by_Lai_Yi_Ohlsen_720p.mp4
added QmWn3YAfxS8kLDgzGUQae1P84DpDKMzLXDr4zVsQUTSoKY 2019/720p/Our_Networks_2019_Regulationism_And_The_Need_For_A_Digital_Postmodernism_by_Corey_Abell_720p.mp4
added QmZTwNKy4RytKrtzYGB5FFpvwjProJZpHAYM2tG24dfrSd 2019/720p/Our_Networks_2019_Remaking_The_Web_Competing_Visions_Of_Decentralization_by_Jack_Jamieson_720p.mp4
added QmQuzgTJFeya8mJ6yozc5ndowDxvysxjWAnQgMHWB2eSMN 2019/720p/Our_Networks_2019_Social_Artistic_And_Theoretical_Experiments_With_Decentralized_Festivals_by_Andreas_Dzialocha_720p.mp4
added QmR5pK74rF2UK9MScWc5DbzAZYEsv54CxmZuqk5vbKafBT 2019/720p/Our_Networks_2019_Solar_Powered_Website_by_Marie_Otsuka_and_Lauren_Traugott-Campbell_720p.mp4
added QmdVMrd5ApoukLvYdumpdc7PJAnuxQwpKnkSz5N7UNrdcv 2019/720p/Our_Networks_2019_Spectacular_Grammar_Infrastructure_As_A_Universal_Language_by_Renee_Reizman_720p.mp4
added QmepRvy5jgM3cNrufoTD6evkYYkWk2ppX9C2H2J9zVk7ue 2019/720p/Our_Networks_2019_The_Impotent_Medium_by_Farhad_Bahram_720p.mp4
added QmVgtgbMNj39pKMBaL3NGJYL3Ti2acn3CFLwRWepKcBdcN 2019/720p/Our_Networks_2019_The_Netifesto_by_Ryan_Kelln_720p.mp4
added zb2rhgPR6VGkFvKZWSj3qyuZkccHYYCiMuVV3tcuyfQzqYUGm 2019/descriptions/Our_Networks_2019_A_Digital_Bundle_Protecting_And_Promoting_Indigenous_Knowledge_Online_by_Dr_Jennifer_Wemigwans.txt
added zb2rhnVcADmTyjTNU2qg7BNStjPtfw9aU3H9GBy5bBvx2ZDFr 2019/descriptions/Our_Networks_2019_A_Technical_Introduction_To_IPFS_by_Hector_Sanjuan.txt
added zb2rhh1bs5XjBosrSKYuUiSeWyg4SBJMXk9bCCdG1CcoGqDq1 2019/descriptions/Our_Networks_2019_Building_A_Modern_Memex_by_Andrew_Louis.txt
added zb2rhbaaCTDmw2sfqQaSn3f5zbDGVi3pCNnWiqtj7yYqSBGYm 2019/descriptions/Our_Networks_2019_Channel_Channel_by_Biyi_Wen.txt
added zb2rhn1THtZUm7ojstCDHGDoL4ZxZRaDd3DgafWwVYFaJUh5m 2019/descriptions/Our_Networks_2019_Collective_Resistance_To_State_Controls_On_Information_by_Cecylia_Bocovich.txt
added zb2rhd8WKRckUeTXiykQLsThJ5zGZaUwCTvudnv38JNrv8E3V 2019/descriptions/Our_Networks_2019_Cryptoeconomics_Towards_Mass_Behavioral_Engineering_Or_A_Network_Commons_by_Jaya_Klara_Brekke_and_Dan_Hassan.txt
added zb2rhcycnvS2b2Ps3mnBxhSZAk7o4Gcu5sJbKAvpx6trzdUtW 2019/descriptions/Our_Networks_2019_DHT_BFT_OMG_Know_Your_Building_Blocks_For_Decentralized_Applications_2019_Edition_by_Kirill_Pimenov.txt
added zb2rhZCAEVxqavS2XZ9hhEw5kpYZCaqeMbtC7wQdgm6tuMWX6 2019/descriptions/Our_Networks_2019_Distributed_Internet_Archiving_Plus_Low_Power_Long_Distance_Meshing_With_LoRaWAN_by_Nick_Sweeting.txt
added zb2rhZCAEVxqavS2XZ9hhEw5kpYZCaqeMbtC7wQdgm6tuMWX6 2019/descriptions/Our_Networks_2019_Distributed_Secure_Routing_In_Permissionless_Flat_Networks_by_Benedict_Lau.txt
added zb2rhZCAEVxqavS2XZ9hhEw5kpYZCaqeMbtC7wQdgm6tuMWX6 2019/descriptions/Our_Networks_2019_Distributing_Knowledge_Of_Digital_Workflows_Via_Workshops_And_Support_Groups_by_Zack.txt
added zb2rhZCAEVxqavS2XZ9hhEw5kpYZCaqeMbtC7wQdgm6tuMWX6 2019/descriptions/Our_Networks_2019_Hugo_Wasm_Plus_IPFS_Blogging_by_Jim_Pick.txt
added zb2rhh5h6jVcGoJt5RdBZxBijn3tGJCtNvXhLpRAxfMdtTMMT 2019/descriptions/Our_Networks_2019_May_First_Technology_And_Revolution_by_Alfredo_Lopez_and_Ellie_Ade_Kur_and_Lorraine_Chuen_and_Victoria_Barnett.txt
added zb2rhhDZR17MekCbpqboN2aaXMLRjGmTno4iaBFWk84FUG5Bb 2019/descriptions/Our_Networks_2019_Mesh_Lightning_Talks_Freifunk_by_Felix_Kaechele.txt
added zb2rhhDZR17MekCbpqboN2aaXMLRjGmTno4iaBFWk84FUG5Bb 2019/descriptions/Our_Networks_2019_Mesh_Lightning_Talks_I2P_Intro_For_App_Devs_by_Idk.txt
added zb2rhY4bPGTB52YdQNs6R8t5ddvg9yiqG591Fdh2JMXLahkJR 2019/descriptions/Our_Networks_2019_Mesh_Lightning_Talks_NYC_Mesh_by_Jillian_Murphy_and_Scott_Rasmussen.txt
added zb2rhnxBmSX7w6xFqmtkY4v8qx7HFTVkmhGHVjwCeaZd3iyfx 2019/descriptions/Our_Networks_2019_Mesh_Lightning_Talks_Sarantaporogr_by_Vassilis_Chryssos.txt
added zb2rhknoniN34TzwX7SckzVw35wc864rgPYc1BPhRLLPW8P51 2019/descriptions/Our_Networks_2019_Networks_As_Sculpture_by_Lai_Yi_Ohlsen.txt
added zb2rhbJMVJZvuqJY9kKYMkEGUKT8Lshj2xXGE8fBsWsXHb17i 2019/descriptions/Our_Networks_2019_Regulationism_And_The_Need_For_A_Digital_Postmodernism_by_Corey_Abell.txt
added zb2rhaNFn8b2JbBePHy31JShzQy7reJvevP49m7hmkvuvLxTi 2019/descriptions/Our_Networks_2019_Remaking_The_Web_Competing_Visions_Of_Decentralization_by_Jack_Jamieson.txt
added zb2rhZaET7VvUJ8eYVAx6zBXzc2F615aBZbgmxVQQWCkzXCsF 2019/descriptions/Our_Networks_2019_Social_Artistic_And_Theoretical_Experiments_With_Decentralized_Festivals_by_Andreas_Dzialocha.txt
added zb2rhmDD38gj3dTLsnrVE2WpfgfX344SPLgP6i3yiF53EhxdZ 2019/descriptions/Our_Networks_2019_Solar_Powered_Website_by_Marie_Otsuka_and_Lauren_Traugott-Campbell.txt
added zb2rhd6aZ2aBSZdbtjcSAEcyv2HnMRcV39KxJHeqHyRwpEvep 2019/descriptions/Our_Networks_2019_Spectacular_Grammar_Infrastructure_As_A_Universal_Language_by_Renee_Reizman.txt
added zb2rhkS1VorDsGoetk8DUsJAjfa5Cu6twGPryTbGwmrwssBQJ 2019/descriptions/Our_Networks_2019_The_Impotent_Medium_by_Farhad_Bahram.txt
added zb2rhZCAEVxqavS2XZ9hhEw5kpYZCaqeMbtC7wQdgm6tuMWX6 2019/descriptions/Our_Networks_2019_The_Netifesto_by_Ryan_Kelln.txt
added QmRB29rYroRmLDnq5NiuFjkUY2NN2XAY3u4SXFs3jNodhd 2019/1080p
added QmaNaRSaDScipAsWRpxBoRFfkoT5YNxKCqYSEQLRVxXdwG 2019/720p
added QmYuMb2vNiwBKwiLYNxDuMmtEzT1BDRUSxNA1WcFErzxxz 2019/descriptions
added QmaWc83FvLkyvw8p7Q45prW7HTXAaLgk8jxmkdciokaE1K 2019
 62.68 GiB / 62.68 GiB [=============================================================================================================================] 100.00%

ipfs.io

In a screen, on freenode/#ipfs-pinbot:

07:55 < benhylau> !pin QmaWc83FvLkyvw8p7Q45prW7HTXAaLgk8jxmkdciokaE1K "2019.ournetworks.ca Recorded Talks"
07:55 < pinbotX> Cluster-pinning via /dns4/cluster2.fsn.dwebops.net/tcp/9094
07:55 < pinbotX> QmaWc83FvLkyvw8p7Q45prW7HTXAaLgk8jxmkdciokaE1K: operation submitted. Waiting for status to reach pinned
...

At one point had to !recover:

08:25 < pinbotX> QmaWc83FvLkyvw8p7Q45prW7HTXAaLgk8jxmkdciokaE1K: an error happened: error has occurred while attempting to reach status: pinned. You can attempt recovery with
                 !recover <cid>.
11:42 < benhylau> !recover QmaWc83FvLkyvw8p7Q45prW7HTXAaLgk8jxmkdciokaE1K
11:42 < pinbotX> Recovering pin via /dns4/cluster2.fsn.dwebops.net/tcp/9094
11:42 < pinbotX> Recover operation triggered for QmaWc83FvLkyvw8p7Q45prW7HTXAaLgk8jxmkdciokaE1K. You can later manually track the status with !status <cid>
11:42 < pinbotX> Status for QmaWc83FvLkyvw8p7Q45prW7HTXAaLgk8jxmkdciokaE1K:
11:42 < pinbotX>   - cluster2 : pinning |
11:42 < pinbotX>   - cluster3 : pin_queued |
11:42 < pinbotX>   - cluster0 : remote |
11:42 < pinbotX>   - cluster4 : pinning |
11:42 < pinbotX>   - cluster1 : remote |

But eventually !status shows pinned on 3 clusters:

15:36 < benhylau> !status QmaWc83FvLkyvw8p7Q45prW7HTXAaLgk8jxmkdciokaE1K
15:36 < pinbotX> Status for QmaWc83FvLkyvw8p7Q45prW7HTXAaLgk8jxmkdciokaE1K:
15:36 < pinbotX>   - cluster2 : pinned |
15:36 < pinbotX>   - cluster3 : pinning |
15:36 < pinbotX>   - cluster0 : remote |
15:36 < pinbotX>   - cluster4 : pinned |
15:36 < pinbotX>   - cluster1 : remote |
16:01 < benhylau> !status QmaWc83FvLkyvw8p7Q45prW7HTXAaLgk8jxmkdciokaE1K
16:01 < pinbotX> Status for QmaWc83FvLkyvw8p7Q45prW7HTXAaLgk8jxmkdciokaE1K:
16:01 < pinbotX>   - cluster3 : pinned |
16:01 < pinbotX>   - cluster0 : remote |
16:01 < pinbotX>   - cluster4 : pinned |
16:01 < pinbotX>   - cluster1 : remote |
16:01 < pinbotX>   - cluster2 : pinned |

Gateway: https://ipfs.io/ipfs/QmaWc83FvLkyvw8p7Q45prW7HTXAaLgk8jxmkdciokaE1K/

ipfs.infura.io

Use Infura API pin_add:

$ curl "https://ipfs.infura.io:5001/api/v0/pin/add?arg=/ipfs/QmaWc83FvLkyvw8p7Q45prW7HTXAaLgk8jxmkdciokaE1K&recursive=true&progress=true"
{"Pins":null,"Progress":40}
{"Pins":null,"Progress":44}
...

The call will timeout after a while, but the pinning seems to be happening in the background. Make subsequent calls later to check on progress:

$ curl "https://ipfs.infura.io:5001/api/v0/pin/add?arg=/ipfs/QmaWc83FvLkyvw8p7Q45prW7HTXAaLgk8jxmkdciokaE1K&recursive=true&progress=true"
{"Pins":null,"Progress":634}
...

Gateway: https://ipfs.infura.io/ipfs/QmaWc83FvLkyvw8p7Q45prW7HTXAaLgk8jxmkdciokaE1K/

Infura pinning never completed, progress seems to get stuck around 3xxx and returns inconsistent results on repeated API calls :(

@benhylau
Copy link
Member

benhylau commented Dec 3, 2019

Internet Archive upload

Archive description:

Recorded talks from State of Our Networks 2019: RE: Infrastructures, held September 20–22, 2019 in Toronto, ON:

- A digital bundle protecting and promoting indigenous knowledge online by Dr. Jennifer Wemigwans
- A technical introduction to IPFS by Hector Sanjuan
- Building a modern Memex by Andrew Louis
- Channel Channel by Biyi Wen
- Collective resistance to state controls on information by Cecylia Bocovich
- Cryptoeconomics: Towards mass behavioral engineering or a network commons by Jaya Klara Brekke / Dan Hassan
- DHT, BFT, OMG: Know your building blocks for decentralized applications (2019 edition) by Kirill Pimenov
- Distributed internet archiving! + Low-power long-distance meshing with LoRaWAN by Nick Sweeting
- Distributed secure routing in permissionless flat networks by Benedict Lau
- Distributing knowledge of digital workflows via workshops and support groups by Zack
- Hugo, WASM plus IPFS blogging by Jim Pick
- May First, technology and revolution by Alfredo Lopez / Ellie Ade Kur / Lorraine Chuen / Victoria Barnett
- Mesh lightning talks: Freifunk by Felix Kaechele
- Mesh lightning talks: I2P intro for app devs by idk
- Mesh lightning talks: NYC Mesh by Jillian Murphy / Scott Rasmussen
- Mesh lightning talks: Sarantaporo.gr by Vassilis Chryssos
- Networks as sculpture by Lai Yi Ohlsen
- Regulationism and the need for a digital postmodernism by Corey Abell
- Remaking the web: Competing visions of decentralization by Jack Jamieson
- Social, artistic, and theoretical experiments with decentralized festivals by Andreas Dzialocha
- Solar powered website by Marie Otsuka / Lauren Traugott-Campbell
- Spectacular grammar: Infrastructure as a Universal language by Renée Reizman
- The impotent medium by Farhad Bahram
- The Netifesto by Ryan Kelln

https://ournetworks.ca/

URL: https://archive.org/details/ournetworks2019

@benhylau
Copy link
Member

benhylau commented Dec 16, 2019 •
edited
Loading

IPFS pinning livestream archive

I assume live.mesh.world already has all the livestream chunks on the m3u8 files in ournetworks/ournetworks.ca/pull/154/ pinned to its IPFS node. So I generated a list of the CID hashes using:

$ grep 'ipfs/' live-*.m3u8 | sed 's|.*ipfs/||g' > ipfs-cids.txt

Attached here: ipfs-cids.txt

ipfs.infura.io

There are 1449 CIDs. Then in a screen, I cat the file and run Infura's pinning API on each CID (takes > a day to iterate through) using:

$ cat ipfs-cids.txt | xargs -I {} curl "https://ipfs.infura.io:5001/api/v0/pin/add?arg=/ipfs/{}&recursive=true&progress=true" >> ipfs-cids.log

The log file gets mostly {"Pins":null} with occasional successful pins like:

...
{"Pins":null,"Progress":77}
{"Pins":["QmUCqFERhFg5EEuTvzBnowVYKEvKuxgsBS4qLpiifcP1jn"]}

So I assume the {"Pins":null} means the Infura node is unable to resolve the CID. Subsequent calls occasionally makes it pin successfully, but more often it still returns {"Pins":null} after many attempts.

After one iteration over the list, 142 of the 1449 CIDs have been pinned onto Infura:

$ grep '\[' ipfs-cids.log | wc -l
142

After 3 iterations we have 542 of the 1449 CIDs pinned. Another couple iterations and having the CIDs pinned on ipfs.io seems to help and finally all 1449 CIDs are pinned 🎉

ipfs.io

Connect live.mesh.world (origin of CIDs) to IPFS Cluster for faster resolutions:

$ ipfs swarm connect /dnsaddr/ipfs-nodes.cluster.ipfs.io

In a screen, use ipfs-cluster-ctl from dist.ipfs.io and pin each CID with:

$ cat ipfs-cids.txt | xargs -I {} ipfs-cluster-ctl --host /dnsaddr/cluster.ipfs.io --basic-auth 'USERNAME:PASSWORD' pin add --name '2019.ournetworks.ca Live Stream Archive' --no-status {}
...
Qmd3ShNn2hZWgWq3AFixpevz2pHF3HfHHEtHTdjdXSPQKa | 2019.ournetworks.ca Live Stream Archive | PIN | Repl. Factor: 2--3 | Allocations: [QmSQvhfz4wQnaDET33qQWZ8yUemLskNs2JeAAydpGLsGN9 QmYAajUVaFMw7EyUsZqwDhbNmCsP8L7VDRLuXNkEw6DCC1 QmaHvxFk6DoNsRHqe2a7UJH66AjGDKPG2HCBxr25YYop32] | Recursive | Metadata: no

Check on pin status using:

$ cat ipfs-cids.txt | xargs -I {} ipfs-cluster-ctl --host /dnsaddr/cluster.ipfs.io --basic-auth 'USERNAME:PASSWORD' status {}

You should find the CID is either pinned or queued for pinning:

QmUCqFERhFg5EEuTvzBnowVYKEvKuxgsBS4qLpiifcP1jn :
    > QmPDJG7Lw7b7QuyR5MtTTHToewfRcAWWBN6Nw58wcog8uW : REMOTE | 2019-12-17T09:05:07.171084983+01:00
    > cluster3             : PINNED | 2019-12-17T08:05:07.171650575Z
    > QmVxomthxuyy6ZYWHofyZd79RNuYHSHzWB2bFmkQcD3SDC : REMOTE | 2019-12-17T09:05:07.171084983+01:00
    > cluster4             : PINNED | 2019-12-17T08:05:07.172763216Z
    > cluster1             : PINNED | 2019-12-17T08:05:07.171496289Z
...
Qmd3ShNn2hZWgWq3AFixpevz2pHF3HfHHEtHTdjdXSPQKa :
    > QmPDJG7Lw7b7QuyR5MtTTHToewfRcAWWBN6Nw58wcog8uW : REMOTE | 2019-12-17T09:03:39.354466884+01:00
    > cluster3             : PIN_QUEUED | 2019-12-17T07:59:28.741299319Z
    > QmVxomthxuyy6ZYWHofyZd79RNuYHSHzWB2bFmkQcD3SDC : REMOTE | 2019-12-17T09:03:39.354466884+01:00
    > cluster4             : PIN_QUEUED | 2019-12-17T07:59:28.779518184Z
    > cluster1             : PIN_QUEUED | 2019-12-17T07:59:28.827876483Z

All 1449 CIDs are pinned when I checked back after a couple hours 🎉

@benhylau benhylau mentioned this issue Dec 17, 2019
Merged
@dcwalk
Copy link
Member Author

dcwalk commented Jan 6, 2020

This was done and the videos are added! Closing this :)

@dcwalk dcwalk closed this as completed Jan 6, 2020
@benhylau benhylau mentioned this issue Apr 11, 2020
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
post-event stream
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@benhylau @dcwalk @Shrinks99