Raw training data

[gcp]

http://leela.online-go.com/training/

License: Public Domain
Hosting sponsored by OGS.

[zhu-jz]

thank you

[Ttl]

I have serious problems with downloading the files. I have tried to download the 92c658d7.zip for several times and each time the download gets interrupted after few minutes. I tried with both Firefox and Chrome but neither seem to be able to download it. Neither browser is able to resume the download and I must start the download from the beginning after failure. The same issue happened earlier with the other files but I was able to download them after retrying few times.

[Marcin1960]

Perhaps FTP would be better? There are nice freeware clients that will continue broken downloads and have various options.

[gcp]

FTP is a far, far worse option as its much less reliable through firewalls. HTTP supports resuming just as well. There's a reason FTP is not used anywhere any more.

Unfortunately I can't really do much here. They are on Google Cloud storage buckets. It's possible downloading them with gsutil allows resuming to work, but I'm not sure. I can try splitting up the data in smaller chunks, maybe.

[Prillan]

You can use the -c option with wget to continue a download.

[Marcin1960]

Well, one can try Firefox Add-ons.

[gcp]

Firefox (and Chrome and every browser) can resume HTTP downloads themselves. There must be something with Google's Cloud Platform that breaks HTTP resumes if it doesn't work.

[Marcin1960]

https://cloud.google.com/compute/docs/instances/transfer-files

On Windows workstations, use the WinSCP client to manage files on your instances through a graphical file browser interface.

Use the gcloud command-line tool. The gcloud compute scp command provides SCP file transfer, creating a SSH key pair for you the first time you connect. Your private key is stored on your local device and its corresponding public key is copied to project or instance metadata.

Open an SFTP connection in your native file browser. On many Linux and macOS systems, the native file browser can connect to the home directory on your instance.

Use the SCP command-line tool. This command works similarly to the gcloud compute scp, but requires you to manually manage your SSH keys.

[marcocalignano]

I have a little question. I am running "dump_supervised all.sgf train.txt" on all.sgf that has 900K games.
Is it normal that it is running since more than 24 Hours wrote more then 12 Gb data and goes over the game over and over again? ( 271169740 positions Writing chunk 16550.)

[gcp]

It passes over the SGF multiple times every time writing out one out of every x positions in a random rotation, so yes.

[isty2e]

During several trials, I constantly fail to download these files. The server doesn't allow me to resume, and the connection is really slow and unstable. Will it be a bad idea to create bittorrent seeds for these?