New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cdn.media.ccc.de mirror #52

Open
lgierth opened this Issue Jan 9, 2016 · 26 comments

Comments

Projects
None yet
@lgierth
Member

lgierth commented Jan 9, 2016

cdn.media.ccc.de is the archive of all video and audio from Chaos Communication Congresses and other events by CCC. Including the 32c3 material, it's about 3 TB.

I made a little test download earlier today from a mirror which is geographically very close to biham (same city!) and got a steady 90 MB/s.

@lgierth lgierth self-assigned this Jan 9, 2016

@lgierth

This comment has been minimized.

Member

lgierth commented Jan 13, 2016

Started mirroring to biham a moment ago: screen bash -c "wget --mirror --no-parent -nv https://ftp.fau.de/cdn.media.ccc.de/ 2>&1 | tee ccc.log"

So fast <3 The mirror I mentioned above is https://ftp.fau.de -- it has plenty more interesting stuff @davidar @achin

@eminence

This comment has been minimized.

Collaborator

eminence commented Jan 13, 2016

while i am achin on IRC, i think maybe you meant @eminence ? 😄

@lgierth

This comment has been minimized.

Member

lgierth commented Jan 13, 2016

Indeed -- totally forgot :)

@davidar

This comment has been minimized.

Member

davidar commented Jan 13, 2016

@lgierth Don't tempt me, I will fill the disk on biham if you're not careful :p

@jbenet

This comment has been minimized.

@lgierth

This comment has been minimized.

Member

lgierth commented Jan 13, 2016

Whoa who initiated that lead? :P

@NDuma NDuma referenced this issue Jan 25, 2016

Closed

Feedback #7

@davidar davidar added the in progress label Jan 29, 2016

@lgierth lgierth removed their assignment Sep 1, 2016

@lgierth

This comment has been minimized.

Member

lgierth commented Sep 15, 2016

I've asked media.ccc.de for rsync access to get this going

@lgierth

This comment has been minimized.

Member

lgierth commented Sep 15, 2016

I've asked media.ccc.de for rsync access to get this going

that was super fast -- I got rsync access and will start mirroring in the next couple of days

@lgierth

This comment has been minimized.

Member

lgierth commented Nov 7, 2016

I finally did an rsync, took about a day at 50 MB/s and it's 3.68 TB in total. Started adding:

> screen sh -c 'go-ipfs/ipfs add -r cdn.media.ccc.de/ | tee ipfs.log'
@Kubuxu

This comment has been minimized.

Member

Kubuxu commented Nov 7, 2016

It might be wise to add it file by file, keep track of the added hashes. From the past ipfs add is bound it fail.

@lgierth

This comment has been minimized.

Member

lgierth commented Nov 7, 2016

It's at 53 % added now, has started really fast with >100 MB/s, and has slowed down over time to ~ 2 GB / min. CPU time, goroutines, and file descriptors are stable, memory has slightly increased over time to 2.3 GB, but that's the same ballpark as e.g. the gateways at the moment.

@lgierth

This comment has been minimized.

Member

lgierth commented Nov 7, 2016

It's at 66 % now, so it took 4 hours for 13 %.

@lgierth

This comment has been minimized.

Member

lgierth commented Nov 7, 2016

Looks pretty good so far, it's still getting gradually slower, but it looks like it'll complete some time tonight. It's at 82 % now and likely not run out of memory.

@lgierth

This comment has been minimized.

Member

lgierth commented Nov 8, 2016

It's finished: https://ipfs.io/ipfs/QmXkqTFfTeZHTiMxPSBwcLWem5cQ8MsmbYfQRQd8Jydyg5

Symlinks don't work yet, we need a patch for ipfs add for that.

@lgierth lgierth self-assigned this Nov 8, 2016

@lgierth

This comment has been minimized.

Member

lgierth commented Dec 26, 2016

New hash, from a sync two days ago: QmXd2t4WbhpDf643ija6byLE4q3L8GBQ3u773wWh5zVRT4

https://ipfs.io/ipfs/QmXd2t4WbhpDf643ija6byLE4q3L8GBQ3u773wWh5zVRT4

@lgierth

This comment has been minimized.

Member

lgierth commented Dec 26, 2016

Symlinks are worked on by @kevina at the moment in ipfs/go-ipfs#3508

@interfect

This comment has been minimized.

interfect commented Dec 31, 2016

So given that https://ipfs.io/ipfs/QmXd2t4WbhpDf643ija6byLE4q3L8GBQ3u773wWh5zVRT4/congress/33c3 doesn't work due to not being able to read symlinks, where do I go to find the actual 33c3 videos not through a symlink?

@jedahan

This comment has been minimized.

jedahan commented Dec 31, 2016

 ▲ ipfs cat /ipfs/QmXd2t4WbhpDf643ija6byLE4q3L8GBQ3u773wWh5zVRT4/congress/2016/dummy.txt
Obviously, there are no recordings yet.

This is a just dummy file to help the mirror admins verify that their
synchronization setup is working as intended. It will disappear before 33c3.
@lgierth

This comment has been minimized.

Member

lgierth commented Jan 2, 2017

@jedahan this mirror isn't contiuously updating itself yet, that's why there's no files for 33c3 yet.

@flyingzumwalt flyingzumwalt added backlog and removed in progress labels Jan 15, 2017

@lgierth

This comment has been minimized.

Member

lgierth commented Jan 30, 2017

Did an rsync run today, it's about 4.5 TiB now, and in the process of adding:

20:17:24 I: Started ./ipfs add -r cdn.media.ccc.de/
20:17:25 O: added QmSeCGfgvjdU28X4k4WJ484cNiBKeczk9uRNw5U2FVJfqF cdn.media.ccc.de/INDEX
@chpio

This comment has been minimized.

chpio commented Mar 13, 2017

@lgierth do you already (:D) have the new hash?

@lgierth

This comment has been minimized.

Member

lgierth commented Jan 4, 2018

New up-to-date mirror, now 5,4 TiB: ipfs://QmW84mqTYnCkRTy6VeRJebPWuuk8b27PJ4bWm2bL4nrEWb -- see #165

@lgierth

This comment has been minimized.

Member

lgierth commented Feb 10, 2018

@TheUser

This comment has been minimized.

TheUser commented Feb 19, 2018

@lgierth I downloaded https://cdn.media.freifunk.net, actually I downloaded it from https://media-mirror.hamburg.freifunk.net/ as it was faster for me.
As I do not have a stable 24/7 IPFS node running at the moment the files is only on my laptop so if someone else can pin them that would be great.

Or even better might be if someone else also downloaded the files and verified my hash, total size is around 26 GB.

https://ipfs.io/ipfs/QmVbm2ZXeZYFVTojsjbsSyELQfpSiWscWaRsgs6s4kcyNj

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment