Raw training data #167

gcp · 2017-11-24T13:03:43Z

License: Public Domain
Hosting sponsored by OGS.

zhu-jz · 2017-11-24T13:13:34Z

thank you

Ttl · 2017-12-04T15:25:24Z

I have serious problems with downloading the files. I have tried to download the 92c658d7.zip for several times and each time the download gets interrupted after few minutes. I tried with both Firefox and Chrome but neither seem to be able to download it. Neither browser is able to resume the download and I must start the download from the beginning after failure. The same issue happened earlier with the other files but I was able to download them after retrying few times.

Marcin1960 · 2017-12-04T15:37:58Z

Perhaps FTP would be better? There are nice freeware clients that will continue broken downloads and have various options.

gcp · 2017-12-04T15:44:26Z

FTP is a far, far worse option as its much less reliable through firewalls. HTTP supports resuming just as well. There's a reason FTP is not used anywhere any more.

Unfortunately I can't really do much here. They are on Google Cloud storage buckets. It's possible downloading them with gsutil allows resuming to work, but I'm not sure. I can try splitting up the data in smaller chunks, maybe.

Prillan · 2017-12-04T15:47:53Z

You can use the -c option with wget to continue a download.

Marcin1960 · 2017-12-04T15:53:52Z

Well, one can try Firefox Add-ons.

gcp · 2017-12-04T16:15:11Z

Firefox (and Chrome and every browser) can resume HTTP downloads themselves. There must be something with Google's Cloud Platform that breaks HTTP resumes if it doesn't work.

Marcin1960 · 2017-12-04T17:50:37Z

https://cloud.google.com/compute/docs/instances/transfer-files

On Windows workstations, use the WinSCP client to manage files on your instances through a graphical file browser interface.

Use the gcloud command-line tool. The gcloud compute scp command provides SCP file transfer, creating a SSH key pair for you the first time you connect. Your private key is stored on your local device and its corresponding public key is copied to project or instance metadata.

Open an SFTP connection in your native file browser. On many Linux and macOS systems, the native file browser can connect to the home directory on your instance.

Use the SCP command-line tool. This command works similarly to the gcloud compute scp, but requires you to manually manage your SSH keys.

marcocalignano · 2017-12-25T20:10:23Z

I have a little question. I am running "dump_supervised all.sgf train.txt" on all.sgf that has 900K games.
Is it normal that it is running since more than 24 Hours wrote more then 12 Gb data and goes over the game over and over again? ( 271169740 positions Writing chunk 16550.)

gcp · 2017-12-26T09:43:30Z

It passes over the SGF multiple times every time writing out one out of every x positions in a random rotation, so yes.

isty2e · 2017-12-31T02:00:51Z

During several trials, I constantly fail to download these files. The server doesn't allow me to resume, and the connection is really slow and unstable. Will it be a bad idea to create bittorrent seeds for these?

killerducky · 2017-12-31T02:26:08Z

Have you tried "wget -c"? (I haven't tried it because I'm able to download the files in one shot.)

zediir · 2018-01-01T10:57:35Z

I'm going to download them and create a torrent as I have around 200Mbps to ogs.

isty2e · 2018-01-01T11:14:12Z

@killerducky Currently the download is ongoing... But it will take at least a week per file, so for the answer to be clear it will take some time.

zediir · 2018-01-01T14:29:29Z

Here's the magnet link

isty2e · 2018-01-01T14:38:49Z

@zediir Thank you for sharing. The bittorrent client is trying to download it, hopefully it will be faster.

zediir · 2018-01-01T15:22:57Z

@isty2e I'm also uploading it to my seedbox so that might make downloading faster. It'll take a while though as I'll have to upload 156GB.

zediir · 2018-01-03T14:42:03Z

@isty2e Managed to get the seedbox working. Your download speed should be somewhat faster now.

isty2e · 2018-01-03T14:46:11Z

@zediir Indeed it is! It was like 35KB/s before and it is much faster now.

zediir · 2018-01-04T21:46:39Z

I'm going to keep uploading the training data to the seedbox and posting (editing this post) updated magnet link to a new torrent when @gcp uploads a new batch of training data.

roy7 · 2018-01-04T21:48:36Z

@isty2e Is your download speed issue that you're in China? If @zediir doesn't already, maybe he or someone else out there would run a seedbox located in China. That might give the best result.

zediir · 2018-01-04T21:50:19Z

Unfortunatly seedbox.io that I use only offers servers from Netherlands, Romania and France. Though @isty2e seems to be downloading much faster (about 2MB/s I think) now that they are downloading from the seedbox.

MartinVingerhoets · 2018-01-04T21:57:21Z

@zediir I downloaded it at 12MB/s (from the Netherlands)

zediir · 2018-01-04T21:59:26Z

Ah that was you :). 12.5MB/s (100Mbps) is the limit for that particular seedbox .

roy7 · 2018-01-04T22:17:46Z

Perhaps @isty2e sets up his own Chinese seedbox to mirror off yours and then downloads from that one once it's all there. :) (Google tells me there are many Chinese seedbox companies.)

isty2e · 2018-01-05T03:32:05Z

Well I am not in China, and my network connection and bandwidth are more than fine. They are exceptionally bad for OGS strangely. Anyway, thanks to @zediir, the training data are all downloaded.

bjiyxo · 2019-04-11T13:58:35Z

@gcp Will you upload training data this week?

Ttl · 2019-04-30T12:17:38Z

leela.online-go.com hasn't worked for a day. Gives error 522 "Connection timed out" for me.

anoek · 2019-04-30T13:46:31Z

@Ttl: There seems to be an issue going on at the provider, I've opened up a support ticket to get the issue resolved.

gcp · 2019-05-02T13:02:38Z

Any progress? The old networks are also offline as they are hosted on the storage server, and that also means I can't clean up those on the live server...

anoek · 2019-05-02T13:20:19Z

I filed a follow up ticket with them yesterday after 24 hours, still haven't heard anything back. Quite frustrating. Needless to say we'll be moving to another provider, but we still need to get access to those files. I'll get another server up today and get you the details so you can at least clean stuff off the live server. For future people googling reliability and support, be warned and stay away from delimiter.com

gcp · 2019-05-03T06:56:09Z

I started re-uploading old networks from my local backup. This is about 108G and it will take me about 2 days.

Re-uploading the >1TB of training data...well you do the math. I will start uploading the new data and hope the old server eventually pops up back long enough so we can transfer from there.

I think we also had the 9x9 networks, the SGF files for matches and self-play on there (just mentioning this for my own reference).

Thanks for the help @anoek. The new server seems to be a lot more responsive too, so yay.

anoek · 2019-05-03T13:02:06Z

Yeah I noticed it was a lot faster as well, so at least that's a win.

Terribly sorry about the problems all. I've never had a service provider just drop a server on the floor and not even respond to any tickets before.

I'll let you know if they do manage to get it back up before you finish uploading everything so we can finish with a server to server transfer.

gcp · 2019-05-05T06:37:53Z

Everything should be back up except for older training data.

bubblesld · 2019-05-13T18:02:42Z

the file train_c9fb22c7.zip uploaded on May 3 is of size 7.4gb.
the same file name is of size 4.6gb on May 5.
Is there something wrong?

Are we going to get new training data this week?

AncalagonX · 2019-05-14T02:56:33Z

When someone has a chance, could we get new Tensorboard logs posted? Right now there aren't any LZ Tensorboard logs available for download.

gcp · 2019-05-14T11:20:03Z

Is there something wrong?

I'll redump and reupload this but that will take a while. Possibly the second dump was missing the part of the data that went out of the training window.

Are we going to get new training data this week?

Uploading should finish in about 10 minutes.

bubblesld · 2019-05-14T11:25:00Z

Is there something wrong?

I'll redump and reupload this but that will take a while. Possibly the second dump was missing the part of the data that went out of the training window.

I have most of the old files. I can upload them if necessary.

Are we going to get new training data this week?

Uploading should finish in about 10 minutes.

Thanks a lot.

gcp · 2019-05-14T13:01:19Z

When someone has a chance, could we get new Tensorboard logs posted? Right now there aren't any LZ Tensorboard logs available for download.

Up in https://sjeng.org/zero

gcp · 2019-05-28T16:22:00Z

the file train_c9fb22c7.zip uploaded on May 3 is of size 7.4gb.
the same file name is of size 4.6gb on May 5.
Is there something wrong?

This file was re-uploaded now.

bubblesld · 2019-08-19T20:24:27Z

Can we also have the update of ac9bcd63? The one uploaded on August 1 should not contain all selfplay from the size of the file.

gcp · 2019-08-20T09:11:43Z

Ok, will check.

l1t1 · 2019-08-20T12:29:59Z

train_a4f5d99a.zip 1433662354 2019-Aug-19 09:06
train_1d9360fd.zip 6657558763 2019-Aug-19 09:04
train_657ae4dd.zip 658146911 2019-Aug-19 09:00
train_fe85a8e4.zip 5849475618 2019-Aug-19 08:59
train_16d9854c.zip 8094154573 2019-Aug-01 07:46
train_ac9bcd63.zip 7188541271 2019-Aug-01 07:35

gcp · 2019-08-23T09:03:25Z

Redumped and re-uploaded ac9bcd63.

bubblesld · 2019-08-24T01:24:26Z

Redumped and re-uploaded ac9bcd63.

many thanks

bubblesld · 2019-09-02T14:42:46Z

It looks like the self-play games from v241 (466fa23a) are missing in the update today?

22nsuk · 2019-12-02T14:53:05Z

@gcp When will it be updated? It's been a while since it wasn't updated.

gcp · 2020-04-20T14:22:35Z

@anoek The server seems to have been down for a few days, can you investigate?

anoek · 2020-04-20T14:31:25Z

@gcp fixed. For some reason the VM was powered off, I'll investigate further. I guess I should have had a heartbeat check on that server..

gcp · 2020-04-20T14:42:25Z

Thanks! Will try to upload missing data overnight.

gcp · 2021-03-03T11:46:11Z

I just finished re-uploading all the files that were lost in one of the server moves/outages. So all the training data is complete now. It will stay up as long as @anoek keeps hosting it.

VivianoRiccardo · 2021-10-07T11:44:38Z

Do these training data show also the output of the Value function of a board position?

Vandertic · 2021-10-07T13:47:34Z

Do these training data show also the output of the Value function of a board position?

I don't think so. Should be just input position (8 moves), color of the current player, outcome of the game and visits distribution.

This was referenced Nov 28, 2017

How to find Information about the new best network #78

Open

Is there any way for us to contribute to the automated training? #234

Closed

Ttl mentioned this issue Dec 10, 2017

fp16 training #330

Closed

ihavnoid mentioned this issue Dec 17, 2017

some mean person maybe do bad thing to this project #403

Closed

gcp mentioned this issue May 3, 2019

How to access earlier leela zero network weights and other public domain data #2369

Closed

l1t1 mentioned this issue May 12, 2019

15b training update #2192

Open

sethtroisi mentioned this issue Aug 16, 2019

Would you consider switching to the MIT license? #160

Closed

Raw training data #167

Raw training data #167

Comments

gcp commented Nov 24, 2017 • edited Loading

zhu-jz commented Nov 24, 2017

Ttl commented Dec 4, 2017

Marcin1960 commented Dec 4, 2017

gcp commented Dec 4, 2017 • edited Loading

Prillan commented Dec 4, 2017

Marcin1960 commented Dec 4, 2017

gcp commented Dec 4, 2017 • edited Loading

Marcin1960 commented Dec 4, 2017

marcocalignano commented Dec 25, 2017

gcp commented Dec 26, 2017

isty2e commented Dec 31, 2017

killerducky commented Dec 31, 2017

zediir commented Jan 1, 2018

isty2e commented Jan 1, 2018

zediir commented Jan 1, 2018 • edited Loading

isty2e commented Jan 1, 2018

zediir commented Jan 1, 2018

zediir commented Jan 3, 2018

isty2e commented Jan 3, 2018

zediir commented Jan 4, 2018

roy7 commented Jan 4, 2018

zediir commented Jan 4, 2018

MartinVingerhoets commented Jan 4, 2018

zediir commented Jan 4, 2018 • edited Loading

roy7 commented Jan 4, 2018

isty2e commented Jan 5, 2018

bjiyxo commented Apr 11, 2019

Ttl commented Apr 30, 2019

anoek commented Apr 30, 2019 • edited Loading

gcp commented May 2, 2019

anoek commented May 2, 2019 via email • edited Loading

gcp commented May 3, 2019

anoek commented May 3, 2019

gcp commented May 5, 2019

bubblesld commented May 13, 2019

AncalagonX commented May 14, 2019 • edited Loading

gcp commented May 14, 2019 • edited Loading

bubblesld commented May 14, 2019

gcp commented May 14, 2019

gcp commented May 28, 2019

bubblesld commented Aug 19, 2019

gcp commented Aug 20, 2019

l1t1 commented Aug 20, 2019

gcp commented Aug 23, 2019 • edited Loading

bubblesld commented Aug 24, 2019

bubblesld commented Sep 2, 2019

22nsuk commented Dec 2, 2019

gcp commented Apr 20, 2020

anoek commented Apr 20, 2020

gcp commented Apr 20, 2020

gcp commented Mar 3, 2021

VivianoRiccardo commented Oct 7, 2021

Vandertic commented Oct 7, 2021

gcp commented Nov 24, 2017 •

edited

Loading

gcp commented Dec 4, 2017 •

edited

Loading

gcp commented Dec 4, 2017 •

edited

Loading

zediir commented Jan 1, 2018 •

edited

Loading

zediir commented Jan 4, 2018 •

edited

Loading

anoek commented Apr 30, 2019 •

edited

Loading

anoek commented May 2, 2019 via email •

edited

Loading

AncalagonX commented May 14, 2019 •

edited

Loading

gcp commented May 14, 2019 •

edited

Loading

gcp commented Aug 23, 2019 •

edited

Loading