Support downloads from yadisk. Closes #5 #6

dnovikoff · Oct 14, 2018 · 721181e · 721181e
1 parent b8c5c7f
commit 721181e
Show file tree

Hide file tree

Showing 30 changed files with 635 additions and 151 deletions.
diff --git a/README.md b/README.md
@@ -11,121 +11,11 @@ This package contains functions for parsing/generating:
 2. Client messages
 3. Server messages
 
-## Howto use proxy server
-I've made this small proxy for researching tenhou.net flash client protocol content.
+## Proxy logging server
+See documentation [here](cmd/tenhou-proxy/README.md)
 
-Download application
-```
-go get -u github.com/dnovikoff/tenhou/cmd/tenhou-proxy
-```
-
-Run application
-```
-$GOPATH/bin/tenhou-proxy
-```
-
-Add to your `hosts` file
-```
-127.0.0.1	b.mjv.jp
-```
-
-Login into flash client http://tenhou.net/0/ .
-Application output would look like
-```
-2018/01/27 00:26:42 Started server on addr ':10080'. Sequence id is 'baab75ab'
-2018/01/27 00:26:55 File for new connection is 'baab75ab_0001.log'
-2018/01/27 00:26:55 Error: EOF
-2018/01/27 00:26:55 Error: Read context done
-2018/01/27 00:26:55 Done with 1
-2018/01/27 00:26:55 File for new connection is 'baab75ab_0002.log'
-2018/01/27 00:26:56 Error: EOF
-2018/01/27 00:26:56 Error: Read context done
-2018/01/27 00:26:56 Done with 2
-2018/01/27 00:26:57 File for new connection is 'baab75ab_0003.log'
-```
-
-Protocol logs will appear in workdir.
-Short example of log result.
-```
-Send: <Z />
-Send: <HELO name="NoName" tid="f0" sx="M" />
-Get: <HELO uname="%4E%6F%4E%61%6D%65" auth="20180127-c5f19a8e" ratingscale="PF3=1.000000&PF4=1.000000&PF01C=0.582222&PF02C=0.501632&PF03C=0.414869&PF11C=0.823386&PF12C=0.709416&PF13C=0.586714&PF23C=0.378722&PF33C=0.535594&PF1C00=8.000000"/>
-Send: <AUTH val="20180127-e3afc0df"/>
-Send: <PXR V="1" />
-Send: <PXR V="1" />
-Get: <LN n="bw1aJ1Pm1y" j="B8C4B11B1D4B8D4D24C3C2C3C1B2C2C" g="o4CM3E1Q12Co4g12BM4M12Q12D1e2P2J2G1G1P1D1G2G"/>
-Send: <Z />
-Get: <LN n="by1aL1Ph1BC" j="B4B4D4B8C3B5B8D4D12B12C3B5C1B4B" g="o4CM3E1Q12Co4k12BM4M12Q12D1e2M2M2G1G1P1D1J2G"/>
-Send: <PXR V="129" />
-```
-
-## pimbooo example server
-Pimboo (Pin-Man-Bamboo) is a small example server for a game of two players.
-The server is created in demonstration purposes only and does not aim for a production.
-
-Download example server
-```
-go get -u github.com/dnovikoff/tenhou/cmd/pimboo-server
-```
-
-Run application
-```
-$GOPATH/bin/pimboo-server
-```
-
-Add to your `hosts` file
-```
-127.0.0.1	b.mjv.jp
-```
-
-1. Login into flash client http://tenhou.net/0/ .
-2. Click on any lobby
-3. See start of the game
-
-![Example of game](https://raw.githubusercontent.com/dnovikoff/tenhou/master/cmd/pimboo-server/example.gif)
-
-The server suggests Ron on any opponent drop and Tsumo on any take.
-If you call a Noten-agari, a Furiten-Ron or agari on a wrong tile, you will pay a penalty.
-
-The game continues until one of the opponents will drop under zero points.
+## Pimbooo example server
+See documentation [here](cmd/pimboo-server/README.md)
 
 ## Tenhou stats and log downloader
-Download tool by typing
-```
-go get github.com/dnovikoff/tenhou/tools/tentool
-```
-
-Initialize the stats repo in current working dir
-```
-./tentool stats init
-```
-
-Download stat files by typing
-```
-./tentool stats download
-```
-The files will be downloaded to ./tenhou/stats folder.
-Repeat this action to get updates.
-
-Initialize the logs repo in current working dir
-```
-./tentool logs init
-```
-
-Collect all log ids from stat files by typing
-```
-./tentool logs update
-```
-Repeat this action, after next download of stats repo
-
-You can download some log archives to reduce download time from here https://yadi.sk/d/D_0tO_135zIKPw
-Place this files into ./tenhou/logs dir and run
-Download log files from collected log ids by typing
-```
-./tentool logs import ./tenhou/logs/file.zip ./tenhou/logs/otherfile.zip
-```
-
-Download log files from collected log ids by typing
-```
-./tentool logs download
-```
+See documentation [here](tools/tentool/README.md)
diff --git a/cmd/pimboo-server/README.md b/cmd/pimboo-server/README.md
@@ -0,0 +1,29 @@
+## pimbooo example server
+Pimboo (Pin-Man-Bamboo) is a small example server for a game of two players.
+The server is created in demonstration purposes only and does not aim for a production.
+
+Download example server
+```
+go get -u github.com/dnovikoff/tenhou/cmd/pimboo-server
+```
+
+Run application
+```
+$GOPATH/bin/pimboo-server
+```
+
+Add to your `hosts` file
+```
+127.0.0.1	b.mjv.jp
+```
+
+1. Login into flash client http://tenhou.net/0/ .
+2. Click on any lobby
+3. See start of the game
+
+![Example of game](https://raw.githubusercontent.com/dnovikoff/tenhou/master/cmd/pimboo-server/example.gif)
+
+The server suggests Ron on any opponent drop and Tsumo on any take.
+If you call a Noten-agari, a Furiten-Ron or agari on a wrong tile, you will pay a penalty.
+
+The game continues until one of the opponents will drop under zero points.
diff --git a/cmd/tenhou-proxy/README.md b/cmd/tenhou-proxy/README.md
@@ -0,0 +1,47 @@
+## Howto use proxy server
+I've made this small proxy for researching tenhou.net flash client protocol content.
+
+Download application
+```
+go get -u github.com/dnovikoff/tenhou/cmd/tenhou-proxy
+```
+
+Run application
+```
+$GOPATH/bin/tenhou-proxy
+```
+
+Add to your `hosts` file
+```
+127.0.0.1	b.mjv.jp
+```
+
+Login into flash client http://tenhou.net/0/ .
+Application output would look like
+```
+2018/01/27 00:26:42 Started server on addr ':10080'. Sequence id is 'baab75ab'
+2018/01/27 00:26:55 File for new connection is 'baab75ab_0001.log'
+2018/01/27 00:26:55 Error: EOF
+2018/01/27 00:26:55 Error: Read context done
+2018/01/27 00:26:55 Done with 1
+2018/01/27 00:26:55 File for new connection is 'baab75ab_0002.log'
+2018/01/27 00:26:56 Error: EOF
+2018/01/27 00:26:56 Error: Read context done
+2018/01/27 00:26:56 Done with 2
+2018/01/27 00:26:57 File for new connection is 'baab75ab_0003.log'
+```
+
+Protocol logs will appear in workdir.
+Short example of log result.
+```
+Send: <Z />
+Send: <HELO name="NoName" tid="f0" sx="M" />
+Get: <HELO uname="%4E%6F%4E%61%6D%65" auth="20180127-c5f19a8e" ratingscale="PF3=1.000000&PF4=1.000000&PF01C=0.582222&PF02C=0.501632&PF03C=0.414869&PF11C=0.823386&PF12C=0.709416&PF13C=0.586714&PF23C=0.378722&PF33C=0.535594&PF1C00=8.000000"/>
+Send: <AUTH val="20180127-e3afc0df"/>
+Send: <PXR V="1" />
+Send: <PXR V="1" />
+Get: <LN n="bw1aJ1Pm1y" j="B8C4B11B1D4B8D4D24C3C2C3C1B2C2C" g="o4CM3E1Q12Co4g12BM4M12Q12D1e2P2J2G1G1P1D1G2G"/>
+Send: <Z />
+Get: <LN n="by1aL1Ph1BC" j="B4B4D4B8C3B5B8D4D12B12C3B5C1B4B" g="o4CM3E1Q12Co4k12BM4M12Q12D1e2M2M2G1G1P1D1J2G"/>
+Send: <PXR V="129" />
+```
diff --git a/tools/tentool/.gitignore b/tools/tentool/.gitignore
@@ -1 +1,2 @@
-tenhou
+tenhou
+export
diff --git a/tools/tentool/README.md b/tools/tentool/README.md
@@ -0,0 +1,53 @@
+## Tenhou stats and log downloader
+Download stat files and log files to work with them on your local computer.
+
+Download tool by typing:
+```
+go get github.com/dnovikoff/tenhou/tools/tentool
+```
+
+Initialize the stats repo in current working dir.
+```
+tentool stats init
+```
+
+I suggest that you first download some archives from my [Yandex.Disk](https://yadi.sk/d/uOv87aVsd-l-3A).
+The files will be downloaded to `./tenhou/stats` folder.
+That could you reduce you downloading time. 
+Do this by typing:
+```
+tentool stats yadisk
+```
+
+Download stat files from tenhou.net.
+The files will be downloaded to `./tenhou/stats` folder.
+Repeat this action when you need to get updates.
+```
+tentool stats download
+```
+
+Initialize the logs repo in current working dir.
+```
+tentool logs init
+```
+
+I suggest that you first download some prebuild zip files with logs from my [Yandex.Disk](https://yadi.sk/d/FIIkaucSNjR3Kw).
+That would be sure times faster, than downloading all logs from tenhou one by one.
+The files will be downloaded to `./tenhou/logs` folder.
+```
+tentool logs yadisk
+```
+
+Collect all log ids from stat files by typing.
+Repeat this action, after next call of `tentool stats download`.
+```
+tentool logs update
+```
+
+Download log files from collected log ids by typing
+```
+tentool logs download
+```
+
+Now you have full database of phoenix logs on your machine.
+Consider reading `tentool stats --help` and `tentool logs --help` on more commands and flags.
diff --git a/tools/tentool/export.json b/tools/tentool/export.json
@@ -1,10 +1,10 @@
 [
     {
         "search": "^2018.+$",
-        "file": "export/2018_part.zip"
+        "file": "export/logs2018_part.zip"
     },
     {
         "search": "^([0-9]{4}).+$",
-        "file": "export/$1.zip"
+        "file": "export/logs$1.zip"
     }
 ]
diff --git a/tools/tentool/logs/cmd.go b/tools/tentool/logs/cmd.go
@@ -5,9 +5,10 @@ import (
 	"os"
 	"path/filepath"
 
-	"github.com/dnovikoff/tenhou/tools/tentool/stats"
-	"github.com/dnovikoff/tenhou/tools/utils"
 	"github.com/spf13/cobra"
+
+	"github.com/dnovikoff/tenhou/tools/tentool/stats"
+	"github.com/dnovikoff/tenhou/tools/tentool/utils"
 )
 
 const (
@@ -105,6 +106,17 @@ func CMD() *cobra.Command {
 	downloadCMD.Flags().BoolVar(&interactiveFlag, "interactive", true, "Use interactive downloader progress.")
 	downloadCMD.Flags().IntVar(&parallel, "parallel", 1, "Number of parallel downloads.")
 
+	yadiskCMD := &cobra.Command{
+		Use:   "yadisk",
+		Short: "Download prebuild log archives from yadisk. This will decrease downloading time.",
+		Run: func(cmd *cobra.Command, args []string) {
+			(&yadisk{
+				interactive: interactiveFlag,
+			}).Run(args)
+		},
+	}
+	yadiskCMD.Flags().BoolVar(&interactiveFlag, "interactive", true, "Use interactive downloader progress.")
+
 	importCMD := &cobra.Command{
 		Use:   "import",
 		Short: "Import zip files with logs",
@@ -157,6 +169,7 @@ func CMD() *cobra.Command {
 	rootCMD.AddCommand(validateCMD)
 	rootCMD.AddCommand(updateCMD)
 	rootCMD.AddCommand(downloadCMD)
+	rootCMD.AddCommand(yadiskCMD)
 	rootCMD.AddCommand(importCMD)
 	rootCMD.AddCommand(exportCMD)
 	rootCMD.AddCommand(collectCMD)

diff --git a/tools/tentool/logs/cmd_collect.go b/tools/tentool/logs/cmd_collect.go
@@ -4,7 +4,7 @@ import (
 	"fmt"
 	"io/ioutil"
 
-	"github.com/dnovikoff/tenhou/tools/utils"
+	"github.com/dnovikoff/tenhou/tools/tentool/utils"
 )
 
 func Collect(args []string) {

diff --git a/tools/tentool/logs/cmd_download.go b/tools/tentool/logs/cmd_download.go
@@ -10,13 +10,14 @@ import (
 	"sync"
 	"time"
 
-	"github.com/dnovikoff/tenhou/tools/utils"
+	"github.com/dnovikoff/tenhou/tools/tentool/utils"
 )
 
 type downloader struct {
 	interactive bool
 	index       *FileIndex
 	parallel    int
+	yadiskURL   string
 }
 
 func (d *downloader) Run() {

diff --git a/tools/tentool/logs/cmd_export.go b/tools/tentool/logs/cmd_export.go
@@ -13,7 +13,7 @@ import (
 	"sort"
 	"strings"
 
-	"github.com/dnovikoff/tenhou/tools/utils"
+	"github.com/dnovikoff/tenhou/tools/tentool/utils"
 )
 
 type ConfigRecord struct {
@@ -56,8 +56,9 @@ func (e *exporter) Run(args []string) {
 		return
 	}
 	for _, v := range mapped.Keys() {
-		_, err := os.Stat(v)
-		if err == nil && !e.force {
+		exits, err := utils.FileExists(v)
+		utils.Check(err)
+		if exits {
 			fmt.Printf("Skipping file %v - already exists\n", v)
 			continue
 		}