This is a reinforcement learning project with pytorch and building the game of Snake. This project is heavily inspired by the freeCodeCamp video: https://www.youtube.com/watch?v=L8ypSXwyBds&t=435s. I am using this tutorial for learning purposes so I will be documenting what I learn in this to make similar projects in the future.
The project functions in 3 parts: 1.) The game which is obviously the snake game, 2.) The model which is the brain meant to understand the game shit, 3.) The agent. Apparently model and agent are different, agent is more like the brain that thinks "what do I think is best" and then the agent is like the decision maker that hears it and chooses the best move to perform. They're closely related but thats just something to keep in mind.
The agent takes in the game and the model. Then comes training that involves many aspects:
- state = current state of the game
- 11 values where 1 means true and 0 means untrue. [danger straight, danger right, danger left, direction left, direction right, direction down, food left, food right, food up, food down]
- action = what move to do next
- [1, 0, 0] = straight, [0, 1, 0] = right turn, [0, 0, 1] = left turn. No backwards because the snake would just kill itself.
- reward, game_over, score = what happens after action
- Reward is +10 = eating food; -10 = game over; else is 0.
- new state = new state after action is made
- remember = store the old state, new state, score, etc.
- model.train() = train the model.
The Q-Value is a quality value which answers if I am in this state and I take this action, how good is that decision? Loss function measures how wrong a model's prediction is. In this project, we use the bellman's equation. This eqution tells you the value of a particular state-action pair should be and based on the reward you just received and the estimated future.
epsilon (ε) = controls randomness or level of trying new actions gamma (γ) = controls how much the agent cares about furture rewards. short memory = agent learns immediately after each move long memory = instead of forgetting expereince, the agent stores them.