What Go players want #1758
Comments
|
Nice!
Here are my 2 cents on the question! |
|
I am a mid-high dan player (4H to professional players), and agree to the points mentioned in OP, maybe except for the exploration part. I think many of them can be addressed in a foreseeable future. For example, the dynamic komi approach will be helpful to estimate the territory difference at least in an approximate manner, and to behave sensibly in lopsided situations. From what I have seen, the most common requests would be an easy installation and an easy-to-use GUI (including play mode and analysis mode). While Lizzie serves the purpose quite well and I am totally fine with it, for example my father in his mid-sixties had some troubles comfortably using it, so I bought a package of Zenith go 7 for him. As @Friday9i mentioned, when handicap go and lower strength play are available, I believe more will be motivated to use LZ, though these are not straightforward to implement. |
|
I am a Go player with little knowledge about programming. Several friends who like play Go including one Go training school teacher expressed their concerns about Leela Zero. Some of the concerns are already included by @yakogo, @Friday9i . I'd like to post the rest.
My personal thinking is for professional players and AI fans, strength is the first priority. For most Go fans, easy to use and more function is their first priority. |
I think these are exactly why we want LZ to be stronger even though no human can beat it. A network is stronger when it could assess more scenarios more accurately. |
|
8d on KGS. *I could write like a 10 page essay on this, and actually started to, but opted to keep it short and sweet. @diadorak humans can still beat it, even on very good hardware. I've beaten it. Stakeout/byflash, who's a very strong korean amateur, has beaten it. Both he and I have even beaten ELF a handful of times (incredibly hard, though. Maybe 1 out of 15-20 games? not counting ladder exploitation). Many pros continue to beat LZ. @Yakago I explore all kinds of things with LZ. I analyze and explore old games and a wide variety of fuseki and joseki positions using LZ. I teach others how to do the same. There's nothing that restricts you to the small handful of in-vogue patterns that bots play against each other and in self-play. Also, LZ provides a wealth of information that is very understandable in human terms. The winrate is also useful information, if a bit unreliable, but don't get too caught up by it; it's just one aspect of what LZ provides. |
|
9D in fox go and tygem. In my opinion, there are 2 most important issues.
Only if these two issues are solved, we can make the Go AI mostly replace the GO teacher and greatly reduce the threshold to learn Go. Otherwise, the Go teacher is still an irreplaceable part when studying Go. |
|
I've been thinking of making a go game using the godot engine for a while, but the project is too big for me to get started. If anyone is interested, here's what I need help with.
If anyone is interested in creating a game development team, please let me know. If we can get a team started, I can get a prototype out quickly. |
|
My two cents here, many people want strong program and it's not irrelevant even though the program superior than top human players. Because many people still want to beat other bots on the net, for fun or for gambling (FoxGo server let users bet on games). It's like chess, even though the programs are already far surpass top human level, people still chase the strongest program. Although it is a little bit ridiculous, it is some kind of truth and I think this demand is great that can't be ignored. |
Unlike the old Go softwares such as leela 10, Zen, crazy stone and etc., that can estimate the value of each move (might not be exactly correct) , the new softwares that improved by self-play do not know how to estimate the true value of each move such as A's value is 10 komi, B's value is 8 komi. When winrate is very high, LZ often select lower value B, so, it is hard to tell which one is the best move.
That would be great. Many of the old softwares such as Leela 10 knows ladder quite well, For those networks such as 10b, 15B that with less potential improvement by selfplay, it will be great to improve them by incorporating human knowledge to deal with those obvious weakness such as does not know ladder. |
|
@dayun110 "Go players want LZ can be run on weak machines, this was proposed by the Go school that has several second hand computers" In place where I live, a computer that can play with 256x20 reasonably well, costs the average monthly pay, and if I made a mistake of buying it, it would be obsolete in a few weeks, because 256x20 will be dumped as 192x15 was. |
This I don't understand. Why wouldn't you be able to run LZ on a weak machine? It will be slower in terms of playouts but still be enormously strong. People have measured it's significantly stronger than Leela 0.11 with as little as 30 playouts, and Leela 0.11 was high amateur dan!
But 256x20 is stronger even if it gets less than half the playouts. What you are saying makes no sense whatsoever. |
|
@gcp "But 256x20 is stronger even if it gets less playouts. What you are saying makes no sense whatsoever." It makes a lot of sense. When I try 256x20 on my PC, it stops to understand ladders - too few playouts in a reasonable time. Either way, for how many people spending their monthly wage or more to buy a very strong PC makes sense? Add to this electricity consumptions by a powerful GPU. |
|
I ran Leela on i3 laptop without any GPU. It it enough to beat me with 10 minutes time control (for her, not for me) even as I am 5d EGF. |
Thanks for writing this. Most of the things asked here make perfect sense and are what I expect, but some of the things people are asking are, I suspect, born out of inexperience in analyzing with computers. This is no surprise, as the Go world had like less than 6 months or so to adapt to the sudden appearance of sufficiently strong computers. In chess they had like 15 years! For example these two: Go players want 'exploration' rather than pure strength You can play out the moves that you like and see what Leela suggests for them, but nothing is ever going to get the computer to suggest weak moves. Because that's just not good. And I think in the long run players who adapt to the principled play will surpass those that do not. If Leela wants to take horribly complicated fights in all your games and you want to play a positional game, you're going to lose out to a player that does take those fights, if that's the best move. One of the revolutions that has happened in chess is that players now rely more (than in the past) on the concrete analysis of variations, rather than general positional guidelines. You can violate every rule you want if your variation works out. And it turns out there's now tools that can tell you that.
Does it not already provide this? You get the score of the best few moves that are considered close enough. If the analysis is not deep enough, you can just play out those moves you are interested in, you don't just have to let it sit idly at the root position. I think this is the most important thing to understand: if you don't get why a move is good or not good, play it out! Play out all the things. Leela never gets tired and will show the refutation of every bad thing you try. You often need to see the concrete variation to understand what happens or why one move can make a difference, but the computer can provide this explanation tirelessly. Edit: This explanation also makes it clear why a good GUI with variation handling is so important. The old Leela GUI can't really do this at all. |
Sure, it'll make a mistake in more situations. It gets weaker with less playouts, this is well understood. But why act as if this is showstopper that makes it impossible to use the program effectively? This attitude makes no sense to me. People have shown that there can still be erroneous ladder analysis with higher playouts and this doesn't stop anyone from using it. I feel that, like life and death (not an issue for LZ, but one for regular Leela), this is something humans "latch on" to because it's something they can understand better, even if it's largely completely irrelevant in the grand scheme of things. It reminds me of the people that kept yelling "computers can't play chess", "computers can't play closed positions" as the Elo of the machines had already surpassed that of the humans quite a bit. There's still a few people saying this, but it's never stopped computers from being an analysis tool. |
|
@dzhurak "I ran Leela on i3 laptop without any GPU. It it enough to beat me with 10 minutes time control (for her, not for me) even as I am 5d EGF." Interesting. What do you mean by "10 minutes time control"? How many playouts Leela gets per move? Which net? |
I have a very related point, even though I don't know if it is even possible to implement this:
3k KGS |
You can just let it run in analysis mode and see if the winrates for 2nd best, 3rd best, ... moves are "close enough" that they are playable, and then play them out. I don't know what relevance the self-play games have with this. The knowledge of the bot generalizes very well outside what it plays in self-play. There's literally nothing that stops you from exploring openings with it right now (unlike Alpha Zero...) |
|
@Marcin1960 I believe dzhurak is referring to old Leela, not Leela zero. When he said 10 minutes, it means Leela (the AI side) use only 10 minutes for a whole game. Based on my experience, what he said is totally true. I have a 5 years old pc, i5cpu and no GPU card. I am kgs 6d and fox 7d, and I lose 9 out of 10 games against it. Also, old Leeza doesn't make ladder mistakes, at least not in 500+ games against me. |
|
@estonte It's interesting project. Sadly I can only help you with point number 1 - 4d EGF(haven't been active tournament player in a while, but still remember how to play the game). So if you decide to go forward with the game and think I'm good enough you can count me up. |
|
@pcengine "I purchased another 1080Ti pc ... Now, my point is, if you really want to improve your go skills, you definitely find this investment worthy." What skills will be improved by such investment? The cheapest 1080Ti PC costs double average monthly wage here. My wife certainly would teach me some new skills after that ;) |
|
OK, I am looking into some cheaper GPUs. What benchmarks matter the most for LZ? Like: |
How do you learn and master some skills in general? My humble speculation is , most people get there by observing, reading, and thinking. LZ on a strong PC not only provides opportunity for you to observe super strong games any time you want, but also shows you many possible moves to read and think. I personally have found it very useful, more useful than the sum of all the go books and lessons that I have ever paid for (which is way more than a top-of-the-line PC). BTW, I don't know if you have taken any pro lessons or not, but we pay around 80 to 150 USD per game to pro players for a teaching game.
Well, if that's the case, how about teach her go? You may end up having a 2080Ti instead of 1080Ti? ;) |
|
As posted above, I'm 4d EGF - been playing go since 2001. Got 1 dan in a little over one year. Achieved that mostly through remembering games of Shusaku. Even though I didn't understand the moves, I just learned them and recreated like 5 hours/day. I don't write it to show off or something, I just want to say that you don't have to understand everything to actually improve. Back to the topic, I expect leela to be a great learning tool. I want to point out few ideas posted here(and not only here, but repeats often in other topics), which might look like a good learning improvement, but in fact are not:
I have to say leela+lizzie is very powerful learning tool as it is - hence the most important in my opinion to use leela as a learning tool is its strength. There is only one thing I would implement, but my programming skills are not yet good enough, so I dunno if this is doable at all. I wonder if it would be possible for leela(or some GUI) to play teaching games. Is it possible to implement parameter like aggression factor? At this moment out of few let's say 53% winrate moves, leela often picks one which leads to complicated fight. It is fine, but teaching wise picking moves which strengthen your own stones might be better. And those moves doesn't weaken the play, cause if you play them out winratio is exactly the same. I actually used leela/lizzie to play with few friends and they said they didn't knew you could play so simple go yet so powerful. |
|
@nukee86 "(Handicap and playing specific level) might be good if you want leela to be a fun toy, but if you want it to be learning tool, I think that's not the most important feature": we agree but ... learning is not the only thing in life, having fun is important too ;-). |
|
@Marcin1960 Only thing that really matters for LZ is raw OpenCL compute power. AMD cards tend to have very good price to performance ratio in that category. I would recommend something in the RX 400 or RX 500 series. There's also the new 2000-series Nvidia cards coming in a month, which have tensor cores (big theoretical speed up) but they are very expensive. |
|
@Friday9i well, we still have gnugo which is 12k I believe? ;) And alot of bots ranked 10k-6d, so if you just wanna play around you don't really need leela zero to do that. I just shared with you my thoughts on the current state of leela used as a learning tool. In my opinion focusing on handicaps or lower level plays might not be very best investment of the resources. But again, that's only if you consider leela as a learning tool. |
|
@nukee86 Yeah for learning, but once again, having fun is important too. |
|
@nukee86 Thanks for the interest. 4 dan is higher than I will ever be. I've never been able to break into single digit kyu :( Im more of a game developer looking to make a teaching tools for others and myself to get better at go. Here's my current game implementation. Any ideas, criticisms, and comments are appreciated. Core Features:
Additional Features: |
|
@estonte It sounds to me like you're trying to re-invent the wheel. Also, what beginner players should look for is the same as what high level players look for: the moves that are strong (which LZ provides), and the moves that they're interested in learning about (which you can use LZ to explore). If you're referring to the concepts and logic related to the moves, that's something that a player can gain only by examining multiple sets of closely related positions, forming hypotheses about the impact of the differences between those positions on the game, testing your hypotheses, observing the results of those tests and drawing conclusions about how they impact your game, and then further refining your hypotheses, aka the scientific method. Either that or ask a strong player who's already done that to just give you the answers (and hope they've gained enough experience and organized their own thoughts well enough to give you a correct and proper explanation). But even then, it doesn't change the fact that you need to do the work yourself, it just means that you'll waste much less time and effort erroneously reinforcing ideas that you misevaluated as being good when they are, in fact, bad. |
|
@Crazyeight101 I think that @estonte want to re-invent the wheel too (I think it is better to add lizzie capabilities to Sabaki, or Sabaki nice interface to Lizzie with a more verbose Leela than starting again from scratch), but he have a good idea: Being able to get from Leela the reasons why a move is preffered from another one. Being able to find and extract 'features' learned and used by Leela through some GTP commands like we are able to do with the heatmap command could be interesting. We could find the 'territory' feature, the 'ladder/ladder breaker' features, perhaps some 'aggressive/passive' feature helping @nukee86 request about selecting moves on their aggressiveness/defensiveness. If we can find where/if Leela has learned theses concepts and be able to 'express' them in a way an analysing tool like Lizzie or any other tool could display it would be a great plus for the Go community. |
|
There's no guarantee that there's any single neuron in the network that is measuring territory, or presence or absence of ladders, or any other "human-friendly" concept. Of course there might be such things, but there might not. Worse, there might be things that look superficially like them but are actually doing something less comprehensible-to-humans (e.g., maybe some set of neurons in some layer is measuring for each point some combination of likely-territory-ness and vulnerability-to-attack, and then you might mistakenly think it's measuring likely-territory-ness and get very confused). I think that if you want LZ to measure something of this sort, it needs to be trained to do so. E.g., one thing I would love to see is a variant of LZ that attempts to estimate a probability distribution over final scores rather than just (as at present) the probability that each player wins. (I suspect this would also enable LZ to play better handicap games and adapt better to different komi.) More radically, one could imagine a version that tries to predict for each point how likely it is to end up as black territory, as white territory, or as neither. (I suspect this would not be valuable in making LZ play better, but it would be a really cool learning/teaching tool.) All these things could in principle be implemented by adding new "heads" to the neural network and training. But I don't think you can usefully hope to get them out of LZ-as-it-is-now without making that sort of change. |
|
I don't think 1 neuron would do it. But if it work simillary than for image recognition, Each layer of the convolutional part of the network grasp even more complex concepts (1st layers detect lines, 2nd level layers detect edges, nth layers detect nose, eyes, ears, nth+1 layer detect faces... see https://www.rsipvision.com/exploring-deep-learning/). I really hope that we can find similar things in Leela networks. |
|
Hey petgo3 at KGS has with 'bm' and 'wr' some rudimental features... |
|
@Cabu Sure, you might get lucky and find human-meaningful features in particular activations of the network. But I don't think you'll ever be able to have confidence that anything you've found actually corresponds to the human-meaningful feature you're interested in. (I do think it would be an interesting project to try to figure out what activations of the various neurons in the network signify, though.) |
|
Training the network is the first step. Visualizing the would be the second step although it is outside of the goals of leelazero for now. I plan to train a much smaller network and visualize the features to find anything meaningful. Visualization of the features would be a start. If we can create a tool for go professionals and go teachers to dissect the networks further than just winrate, we might find human-meaningful features or features that human never discover. Im more of a game developer than a researcher so I just want a fun tool for players to explore how leelazero think. I also don't like sabaki and lizzie because the gui is clunky and has too many options. I think it would be easier and faster to do redo the gui in a game engine. This is more of an artistic nitpicking though. Sabaki and lizzie are great tools. |
|
https://github.com/AncalagonX/leela-zero/releases/tag/LZ-Review-mode-v6__LZv0.16-next looks relevant to some ideas here. |
|
@alreadydone did you fix the problems that komi was having when used with 20b networks? I read the readme, but didn't see anything addressing it. |
|
I tested a438 against ELFv0 -p 1 with 8 stones under handicap mode and the result was B+13.5, so I it at least doesn't behave badly at handicap. The initial komi was ~49 at the better end (pos) and ~214 at the worse end (neg). |
|
@alreadydone I tried your newest release yesterday, and it works perfectly, no problem. Really appreciate your work, special thanks to your very detailed readme file. Would you consider to PR your dynamic komi? I'm sure it's going to be very helpful for lots of goplayers. |
|
It's pull request #1772 but may be too complicated for people to review; I haven't really explained what I've done to the code yet. |
|
Thanks to @alreadydone for mentioning my "LZ Review Mode" branch above (link again here). The "LZ Review Mode" branch forces LZ to show the Lizzie user a much wider range of perfectly acceptable moves for each board position, rather than the very small number of maximally optimal moves which LZ normally prefers to explore. It allows Lizzie to show us a huge variety of perfectly fine fuseki moves (sometimes even very wacky but workable plays) or other moves throughout the game which LZ would have never played on its own. If there is interest in merging this with the official LZ project, I can start a PR for it. |
|
I've also made another modified version of LZ that conducts its MCT search with the goal of forcing every move of the game to a 50% tied winrate, regardless of how optimally/suboptimally the other player is able to play. See the SGFs listed here from my KGS bot "YourRank" for examples of what these game look like. When I used the 15x192 LZ network back in April, this bot was capable of consistently forcing verified professional players and other 9dan players on Tygem to 0.5-point losses. The "50% tied winrate" could conceivably be adjusted for different effects. For example, an LZ which gradually shifts every 15 moves between superhuman punishment of its opponent's moves vs. allowing the player to catch back up to an even position over the next 15 or so moves. Or an LZ which always lets the human win, just barely. Or an LZ which always tries to punish the player only enough to maintain a 60% or 70% winrate. Or an LZ which chooses one of these strategies at random, or which makes calculated suboptimal "mistakes" on random moves. |
|
@AncalagonX Your review mode is great. I can use it for my studying games. It's like taking a multiple choice test in go. There's so many available moves in go that it can get overwhelming. This removed a lot of stress. As for your 50% tied winrate version of lz, is it on github? I'm very much interested in testing and using it. Your other ideas are great too. What about a dynamic winrate system where the player can choose the winrate like 50% or 70%? This way we can use one network with multiple difficulty levels that's different from handicap games. |
|
@AncalagonX what about deciding 50% whether or not the bot will win, on the first move? Then it would seem like the bot is actually your rank, rather than "slightly better than you but impossible to beat" |
|
Are there plans to merge the 50% winrate version into master? I've played around with both the analysis and winrate versions and they are great, would be really nice to have them in master and be able to just run with Sabaki. |
and others
In another issue I saw a comment by @gcp which in part asked the question "What do go players want?"
I thought to myself that here would be an opportunity for me to actually contribute to the project, despite my limited programming knowledge.
I'm a 3d player from europe, and I would like to theorize about some things that I think the major segment of go players ranging from Single-digit kyus to lower dan players would care about with regards to Leela Zero and related tools.
The question was mentioned in the issue about the development of the 40 block networks -> i.e. in context to the ongoing pursuit of stronger and stronger networks
I of course welcome other go players to chip, on what they think is important!
Go players want a strong program, but..
Many people are fascinated by the idea of stronger and stronger programs, and the idea of a pursuit for optimal play. But at some point it starts to become irrelevant. FineArt, ELF, LeelaZero, AlphaGo.. it doesn't really matter - they are all vastly superior to top human professionals, and for the common player, the strength gap between the programs become irrelevant. They're all so strong that we do not even understand why there is a difference between them.
Go players want to analyze their games
This is the most important thing to the majority of players. During the EGC, there were so many people with Lizzie on their laptops. People have made websites for analysis, and people are paying for new computers and computing time on Amazon etc. - The most easy to use and easy to understand tool out there will take the attention from anyone.
Go players want to be able to understand Leela (winrates)
As analysis is so important, it is important that what you get from Leela makes some kind of sense in regards to the game you're playing.
If the program is so strong, that you can't play say a normal corner variation, without the winrate being 90% for one, then analysis becomes too confusing for the player. There needs to be a distinction between what is slightly favorable, and what is very favorable - with respect to a human standard!
Something that would be helpful in this regard, is for example a 'territory estimation'. For instance the evaluation for a given position could say that a player is 99% favored - but because Leela is so strong, the territorial difference might only be 1 or 2 points. This makes things extremely volatile for a human player
Go players want 'exploration' rather than pure strength
This one I could be wrong about, but I believe most people take delight in Go being a game with many options. Analyzing a game, where the program says that there's just a single good move all the time, is quite dull to be honest. Most people would prefer to see several good options if such exist, so they can make an active decision about how they personally want to approach the game. I believe many would trade a few 100 elo for that.
Go players want Leela to know Ladders
And in general to have as little 'obvious' blind spots as possible.
Currently, I believe that it is very hard to put Leela into a bad ladder situation - when Leela is actually playing the game - But when analyzing human games, it is quite easy to find cases where Leela will simply disregard the presence of a ladder, and be 'surprised' with a big jump in evaluation, when the ladder stone is moved. It is not that hard to reproduce for instance with Lizzie, but as an example I will refer to this recent game from the Pair Go World Championship: https://www.youtube.com/watch?v=xuOSfce3Y3w&t=1285s
Try to follow the game with Lizzie if you're curious.
(For reference, I looked with 5-10k playouts, I don't know if there will be a difference with higher numbers, but for most people, they won't expect much better evaluations than that during a review)
Go players want Leela to be sensible in lopsided situation
This one is well known and there is on going development on the subject. Again it is related to the fact that we're analyzing our games. We are only human, and very often we will find ourselves in lopsided situations. Heck, even some of the strongest players we have, will easily find themselves in very lopsided situations if they ask Leela. But to us the situations are not hopeless, as our opponents will make mistakes, and therefore we want a sensible way to look for a comeback.
Go players want moves that are ideal for them
This is probably not something the programming community will solve, but I believe Go players really want to know what kind of moves are fit for them. I believe there is currently a sort of a divide between players that embrace AI and players that shy away from it.
Players are not sure, if the way a computer plays, is something that is objectively good, or just something that is good for a computer.
I believe currently it is teachers and strong players that will have to answer that question, but I guess it gives some insight to how we think about these things :).
I hope some of you will chip in, and that this can be useful for someone who has some ideas.
The text was updated successfully, but these errors were encountered: