Skip to content
This repository has been archived by the owner. It is now read-only.

cdn.media.ccc.de mirror #52

Open
ghost opened this issue Jan 9, 2016 · 26 comments
Open

cdn.media.ccc.de mirror #52

ghost opened this issue Jan 9, 2016 · 26 comments
Labels

Comments

@ghost
Copy link

@ghost ghost commented Jan 9, 2016

cdn.media.ccc.de is the archive of all video and audio from Chaos Communication Congresses and other events by CCC. Including the 32c3 material, it's about 3 TB.

I made a little test download earlier today from a mirror which is geographically very close to biham (same city!) and got a steady 90 MB/s.

@ghost ghost self-assigned this Jan 9, 2016
@ghost
Copy link
Author

@ghost ghost commented Jan 13, 2016

Started mirroring to biham a moment ago: screen bash -c "wget --mirror --no-parent -nv https://ftp.fau.de/cdn.media.ccc.de/ 2>&1 | tee ccc.log"

So fast <3 The mirror I mentioned above is https://ftp.fau.de -- it has plenty more interesting stuff @davidar @achin

@eminence
Copy link
Collaborator

@eminence eminence commented Jan 13, 2016

while i am achin on IRC, i think maybe you meant @eminence ? 😄

@ghost
Copy link
Author

@ghost ghost commented Jan 13, 2016

Indeed -- totally forgot :)

@davidar
Copy link
Collaborator

@davidar davidar commented Jan 13, 2016

@lgierth Don't tempt me, I will fill the disk on biham if you're not careful :p

@ghost
Copy link
Author

@ghost ghost commented Jan 13, 2016

Whoa who initiated that lead? :P

@davidar davidar added the in progress label Jan 29, 2016
@ghost ghost removed their assignment Sep 1, 2016
@ghost
Copy link
Author

@ghost ghost commented Sep 15, 2016

I've asked media.ccc.de for rsync access to get this going

@ghost
Copy link
Author

@ghost ghost commented Sep 15, 2016

I've asked media.ccc.de for rsync access to get this going

that was super fast -- I got rsync access and will start mirroring in the next couple of days

@ghost
Copy link
Author

@ghost ghost commented Nov 7, 2016

I finally did an rsync, took about a day at 50 MB/s and it's 3.68 TB in total. Started adding:

> screen sh -c 'go-ipfs/ipfs add -r cdn.media.ccc.de/ | tee ipfs.log'
@Kubuxu
Copy link
Contributor

@Kubuxu Kubuxu commented Nov 7, 2016

It might be wise to add it file by file, keep track of the added hashes. From the past ipfs add is bound it fail.

@ghost
Copy link
Author

@ghost ghost commented Nov 7, 2016

It's at 53 % added now, has started really fast with >100 MB/s, and has slowed down over time to ~ 2 GB / min. CPU time, goroutines, and file descriptors are stable, memory has slightly increased over time to 2.3 GB, but that's the same ballpark as e.g. the gateways at the moment.

@ghost
Copy link
Author

@ghost ghost commented Nov 7, 2016

It's at 66 % now, so it took 4 hours for 13 %.

@ghost
Copy link
Author

@ghost ghost commented Nov 7, 2016

Looks pretty good so far, it's still getting gradually slower, but it looks like it'll complete some time tonight. It's at 82 % now and likely not run out of memory.

@ghost
Copy link
Author

@ghost ghost commented Nov 8, 2016

It's finished: https://ipfs.io/ipfs/QmXkqTFfTeZHTiMxPSBwcLWem5cQ8MsmbYfQRQd8Jydyg5

Symlinks don't work yet, we need a patch for ipfs add for that.

@ghost ghost self-assigned this Nov 8, 2016
@ghost
Copy link
Author

@ghost ghost commented Dec 26, 2016

New hash, from a sync two days ago: QmXd2t4WbhpDf643ija6byLE4q3L8GBQ3u773wWh5zVRT4

https://ipfs.io/ipfs/QmXd2t4WbhpDf643ija6byLE4q3L8GBQ3u773wWh5zVRT4

@ghost
Copy link
Author

@ghost ghost commented Dec 26, 2016

Symlinks are worked on by @kevina at the moment in ipfs/go-ipfs#3508

@interfect
Copy link

@interfect interfect commented Dec 31, 2016

So given that https://ipfs.io/ipfs/QmXd2t4WbhpDf643ija6byLE4q3L8GBQ3u773wWh5zVRT4/congress/33c3 doesn't work due to not being able to read symlinks, where do I go to find the actual 33c3 videos not through a symlink?

@jedahan
Copy link

@jedahan jedahan commented Dec 31, 2016

 ▲ ipfs cat /ipfs/QmXd2t4WbhpDf643ija6byLE4q3L8GBQ3u773wWh5zVRT4/congress/2016/dummy.txt
Obviously, there are no recordings yet.

This is a just dummy file to help the mirror admins verify that their
synchronization setup is working as intended. It will disappear before 33c3.
@ghost
Copy link
Author

@ghost ghost commented Jan 2, 2017

@jedahan this mirror isn't contiuously updating itself yet, that's why there's no files for 33c3 yet.

@flyingzumwalt flyingzumwalt added backlog and removed in progress labels Jan 15, 2017
@ghost
Copy link
Author

@ghost ghost commented Jan 30, 2017

Did an rsync run today, it's about 4.5 TiB now, and in the process of adding:

20:17:24 I: Started ./ipfs add -r cdn.media.ccc.de/
20:17:25 O: added QmSeCGfgvjdU28X4k4WJ484cNiBKeczk9uRNw5U2FVJfqF cdn.media.ccc.de/INDEX
@chpio
Copy link

@chpio chpio commented Mar 13, 2017

@lgierth do you already (:D) have the new hash?

@ghost
Copy link
Author

@ghost ghost commented Jan 4, 2018

New up-to-date mirror, now 5,4 TiB: ipfs://QmW84mqTYnCkRTy6VeRJebPWuuk8b27PJ4bWm2bL4nrEWb -- see #165

@ghost
Copy link
Author

@ghost ghost commented Feb 10, 2018

@TheUser
Copy link

@TheUser TheUser commented Feb 19, 2018

@lgierth I downloaded https://cdn.media.freifunk.net, actually I downloaded it from https://media-mirror.hamburg.freifunk.net/ as it was faster for me.
As I do not have a stable 24/7 IPFS node running at the moment the files is only on my laptop so if someone else can pin them that would be great.

Or even better might be if someone else also downloaded the files and verified my hash, total size is around 26 GB.

https://ipfs.io/ipfs/QmVbm2ZXeZYFVTojsjbsSyELQfpSiWscWaRsgs6s4kcyNj

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
10 participants