Skip to content
Permalink
Browse files

Adds a "Joseki Explorer" React App (#875)

* start on some react query thing

* next working version

* add table methods, pagination, correct initial heatmap.

* add winrate to database extractor

* add frontend winrate integration, redo reactive spacing, strip logging, revert unneeded custon go board

* fix "noopener", fix margin around pagination buttons

* serve npm built dir, delete dead code

* fix stone padding

* bump version for react-go-board (thanks frank!)

* add message for end-of-sequence.  fix 100% width of winrate chart.

* update joseki gitignore and package-lock file

* update favicons

* add wgo to .gitignore

* fix default index title, update favicon manifest

* migrate all opening related files to oneoffs/joseki, update imports accordingly, add colors to app

* stray sql query output removed

* update readme to remove create-react-app fluff

* add license to file

* unused import

* first round of comments

* more comments.

* track empty next_moves, use flask.request.get_json(), remove unused json dep, fix a typo

* comments
  • Loading branch information...
amj committed Sep 10, 2019
1 parent 9b1d866 commit ad14e92593b4c86330f31af5e58a46d71d295f6a
@@ -10,6 +10,8 @@ pyvenv.cfg
.DS_store
logs/
saved_models/
wgo/*
oneoffs/wgo/*
ratings.db
cbt_ratings.db
.rsync_log
@@ -0,0 +1,24 @@
# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.

# dependencies
/node_modules
/.pnp
.pnp.js

# testing
/coverage

# production
/build
build*

# misc
.DS_Store
.env.local
.env.development.local
.env.test.local
.env.production.local

npm-debug.log*
yarn-debug.log*
yarn-error.log*
@@ -0,0 +1,41 @@
A joseki explorer react app and associated python scripts for populating the
database (`opening_freqs.py`) creating static reports
(`opening_freqs_export.py`) and a flask microservice to serve the app/json.

## Available Scripts (all from create-react-app)

In the project directory, you can run:

### `npm start`

Runs the app in the development mode.<br>
Open [http://localhost:3000](http://localhost:3000) to view it in the browser.

The page will reload if you make edits.<br>
You will also see any lint errors in the console.

### `npm test`

Launches the test runner in the interactive watch mode.<br>
See the section about [running tests](https://facebook.github.io/create-react-app/docs/running-tests) for more information.

### `npm run build`

Builds the app for production to the `build` folder.<br>
It correctly bundles React in production mode and optimizes the build for the best performance.

The build is minified and the filenames include the hashes.<br>

### `npm run eject`

**Note: this is a one-way operation. Once you `eject`, you can’t go back!**

If you aren’t satisfied with the build tool and configuration choices, you can `eject` at any time. This command will remove the single build dependency from your project.

Instead, it will copy all the configuration files and the transitive dependencies (Webpack, Babel, ESLint, etc) right into your project so you have full control over them. All of the commands except `eject` will still work, but they will point to the copied scripts so you can tweak them. At this point you’re on your own.

You don’t have to ever use `eject`. The curated feature set is suitable for small and middle deployments, and you shouldn’t feel obligated to use this feature. However we understand that this tool wouldn’t be useful if you couldn’t customize it when you are ready for it.

### `npm run build` fails to minify

This section has moved [here](https://facebook.github.io/create-react-app/docs/troubleshooting#npm-run-build-fails-to-minify)
@@ -0,0 +1,191 @@
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.


import sys
sys.path.insert(0, '.') # nopep8
import logging
import sqlite3
import collections
import datetime as dt

from flask import Flask, g, jsonify
from timeit import default_timer as timer

import os
import flask

import oneoffs.joseki.opening_freqs as openings

# static_folder is location of npm build
app = Flask(__name__, static_url_path="", static_folder="./build")

# Path to database relative to joseki_query.py, with pre-extracted joseki
# information (see 'opening_frequencies.py')
DATABASE = ''
assert os.path.exists(DATABASE)

def get_db():
db = getattr(g, '_database', None)
if db is None:
db = g._database = sqlite3.connect(DATABASE)
return db

@app.teardown_appcontext
def close_connection(exception):
db = getattr(g, '_database', None)
if db is not None:
db.close()


def get_sequence_hour_counts(seq):
cur = get_db().execute('''
select run, hour, count from joseki_counts
where seq_id in (select id from joseki where seq = ?);
''', (seq, ))
return list(cur.fetchall())

def seq_id_or_none(db, seq):
s_id = db.execute("select id from joseki where seq = ?", (seq,)).fetchone()
return s_id[0] if s_id else None

@app.route("/")
def index():
return app.send_static_file("index.html")

@app.route("/nexts", methods=["POST"])
def next_moves():
d = flask.request.get_json()
prefix = d['params']['prefix']
run = d['params']['run']

db = get_db()

# If error, return this default result.
res = {'count': 0, 'next_moves': {}}

# For the blank board, there's no 'prefix' string sent, and so there's no
# 'next_move' information to be had. This selects and counts all the unique
# opening moves from the joseki table.

# TODO: Once the db is rebuilt with empty opening stats, this special case
# could be removed.
if not prefix:
nexts = db.execute("select distinct(substr(seq, 0, 7)), count(*) from joseki group by 1;").fetchall()
total = sum([n[1] for n in nexts])
else:
s_id = seq_id_or_none(db, prefix)
if not s_id:
return jsonify(res)

if run is None:
nexts = db.execute("""
select next_move, sum(count) from next_moves where seq_id = ? group by 1
""", (s_id,)).fetchall()
total = db.execute("select sum(count) from joseki_counts where seq_id = ?",
(s_id,)).fetchone()[0]
else:
start = timer()
nexts = db.execute("""
select next_move, sum(nm.count)
from next_moves as nm join joseki_counts as jc
on jc.id == nm.joseki_hour_id
where nm.seq_id = ? and jc.run = ? group by 1
""", (s_id, run)).fetchall()
end = timer()
print('%.4f seconds for fancy join.' % (end-start,))
total = db.execute("select sum(count) from joseki_counts where seq_id = ? and run = ?",
(s_id, run)).fetchone()[0]

if not nexts:
print("No next moves found, post params:", d['params'])
return jsonify(res)

next_moves = dict(nexts)
tot = sum(next_moves.values())

for k in next_moves:
next_moves[k] /= tot

max_v = max(next_moves.values())

next_moves = {k:v / max_v for k,v in next_moves.items() if v > 0.001}
res = {'count': total,
'next_moves': next_moves}

return jsonify(res)


@app.route("/games", methods=["POST"])
def games():
d = flask.request.get_json()
prefix = d['params']['sgf']
sort_hour = d['params']['sort']
run = d['params']['run']
# "page" is 1-indexed, so subtract 1 to get the proper OFFSET.
page = d['params']['page'] - 1
db = get_db()

if sort_hour.lower() not in ('desc', 'asc'):
print("Invalid input for sort_hour param: ", sort_hour)
return jsonify({'rows': []})

s_id = seq_id_or_none(db, prefix)
if not s_id:
return jsonify({'rows': []})

q = """select example_sgf, hour, run, b_wins*1.0/count from joseki_counts
where seq_id = ? {} order by hour {} limit 30 offset ?""".format(
"and run = ?" if run else "", sort_hour)

if run:
rows = db.execute(q, (s_id, run, page * 30)).fetchall()
else:
rows = db.execute(q, (s_id, page * 30)).fetchall()

res = [ {'game': os.path.basename(r[0]), 'hour': r[1],
'run': r[2], 'winrate': r[3]} for r in rows]
return jsonify({'rows': res})


@app.route("/search", methods=["POST"])
def search():
d = flask.request.get_json()
print(d)
query = d['params']['sgf']

ts = lambda hr: int(dt.datetime.strptime(hr, "%Y-%m-%d-%H").timestamp())
ranges = openings.run_time_ranges(get_db())
interps = openings.build_run_time_transformers(ranges)

runs = sorted(ranges.keys())

cols = []
cols.append({'id': 'time', 'label': '% of Training', 'type': 'number'})
for run in runs:
cols.append({'id': run + ' count', 'label': run + ' times seen', 'type': 'number'})

sequence_counts = get_sequence_hour_counts(query)

rows = collections.defaultdict(lambda: [0 for i in range(len(runs))])

for r, hr, ct in sequence_counts:
key = interps[r](ts(hr))
idx = runs.index(r)
rows[key][idx] = ct

row_data = [ {'c': [ {'v': key} ] + [{'v': v if v else None} for v in value ] }
for key,value in rows.items()]
obj = {'cols': cols, "rows": row_data, "sequence": query}
return jsonify(obj)
File renamed without changes.

0 comments on commit ad14e92

Please sign in to comment.
You can’t perform that action at this time.