Neckerworld is a computer vision game designed to teach students about human and computer vision systems, object detection and identification, visual field construction, autonomous movement and strategy.
All players and resources in the game are cubes. The cube players are guided solely through autonomous computer vision programs. No human manual control or input is permitted during gameplay.
To successfully play the game requires a competent program to do object detection and identification, playing field knowledge representation and strategic decision making.
The Necker Cube Illusion (first published as a rhomboid in 1832 by Swiss crystallographer Louis Albert Necker) is a well known example of a perceptual illusion where the orientation of the cube is unstable and easily flips between two states. Serious study and research into why we perceive multiple states leads to a set of questions about the very nature of our vision.
Further, cubes are very simple objects which can be used to illustrate many other aspects of human and computer vision, such as the effects of contours, shading, texture, occlusion, size-constancy and more.
Components within the Neckerworld programs allow an instructor to easily construct many of these examples and allows students to learn a lot about visual perception.
Visual Systems are very different. They are uniquely engineered to serve the needs of the organism. They have evolved over 100's of millions of years. The human visual system serves humans well, but not zebras, owls, rats, bees, or spiders.
Three guiding principles of the human visual system:
- Finding water and food to eat
- Don't get eaten
- Find a mate
Consequently, game points are scored through three activities:
- Finding and reaching food/liquid resources
- Killing off predators
- Mating and having child cubes
For a quick demo, check out this video: https://youtu.be/79xiX2FlELk
There are five types of players in the game.
The male, female and enby players are controlled remotely and autonomously via programs created by the game participants. The predators and resources are provided by and controlled by the gaming field server.
An IRL analogy would be a bumper-car ride at an amusement park where we would attach a forward looking camera to the front of each car. Then, by a radio remote-control link, our computer would receive images from the camera and send commands to drive the bumper-car left or right, forward or backward. The bumper-car would then have to pursue or avoid other cars based on color, appearance and game strategy.
Game points are earned in three different ways and determine which player wins the game.
Energy points are the vital resource that keeps a cube alive during the game. Each second of play uses up energy points as does moving around the playing field. Energy points come from feeding on the resource cubes.
A player (Male, Female or Enby) earns game points:
- Feeding on a resource cube. Each hit on a resource cube earns 200 game points and 200 energy points with a maximum of 1000 points.
- Killing off a predator. All the predator's energy points are transferred to the victor as game points.
- When mating half the energy points of the male and female are given to the child. As compensation, those lost energy points are added to the player's game points.
Game play is completely autonomous and a winning program will have fast object detection, an internal representation of the field, a movement strategy that avoids field obstacles and excellent decision making on what targets will yield the most points. A prototype player program is provided (see client/nwplay.py) that is a starter program for competition. The player programs are remote from the game server and connected by a messaging protocol. Consequently, autonomous player programs can be developed on any compute platform desired using favorite systems and tools.
The Neckerworld consists of a playing field server and one or more remote client players. The server places predators and food resources on the field and controls their activity and movement. The client player programs exchange messages (JSON files) with the server:
- Messages to the server request new field images to be returned and specify cube movement and rotation.
- Messages from the server supply current cube status, location, speed, and the cube's-eye view of the playing field.
The client players and programs can be on any computer running any operating system with any programming language.
Follow these steps to create and install the game server:
# If you installed from the github zip file, rename the top-level directory:
mv Neckerworld-main Neckerworld
# Compile the multi-thread server code
cd ~/Neckerworld/server/mtserver/mtserver
make
cd ~/Neckerworld/server
make
# Start the server with a playing-field file
./cube ~/Neckerworld/setup/nwtest.json
# To activate predators on the playing field select the server window and type "S"
# This will enable the server's basic strategy (control of predators and resources).
S
# In a separate window start the player program
# Make sure port 2020 is open and not fire-walled
cd ~/Neckerworld/client
./nwplay.py
A crucial element of this game is the proper identification of cube objects and positions. Each client player needs to scan the environment and identify friends, foes, mates and energy (food) resources. This requires that each received cube's-eye image is run through an object detection and identification program to determine what's in the visual field.
Current solutions for this type of problem use a neural network trained on the visual objects of interest. This repository provides two pre-trained neural network models that may be used for game play. The models run out-of-the-box with a Tensorflow-2 programming environment.
Training a neural network on cube objects is a non-trivial undertaking and requires substantial computer resources. A YOLOv5 model using PyTorch for training is documented in the "train" directory. This runs well on an NVidia GPU system using CUDA. There are multiple utilities for setting up the required image and label files with annotations and bounding boxes. Image samples and a label dataset suitable for training is contained within the training directory and provides an accelerated start to developing a preferred neural network model.
The server has only been tested on Ubuntu 20.04.6 LTS and Ubuntu 22.04.3 LTS. The desktop edition is required as both the server code and player will open a graphical window to function.
The following packages are needed to compile and run these programs:
- python3
- python3-pip
- python3-tk
- python3-opencv
- python3-matplotlib
- libglfw3 libglfw3-dev
- mesa-utils libglu1-mesa-dev mesa-common-dev
- freeglut3 freeglut3-dev
- libglew-dev libglm-dev
- libpng-dev
- make cmake pkg-config
- g++
- imagemagick
- tensorflow
- xlib
- playsound
- torch (for custom model training with YOLOv5)
See script "requirements.sh" for details.
A user has reported that Linux Mint 21.2 Cinnamon ver 5.8.4 also works.
All programs and materials in this repository are under the MIT License. The MIT license is a simple open source license agreement that grants users the right to reuse code for any purpose, including inside proprietary software, which sets it apart from a Copyleft license.
See: https://mit-license.org and https://en.wikipedia.org/wiki/MIT_License for more information.
Discussion and citations about Neckerworld:
- Website - http://www.neckerworld.com
- YouTube Channel - https://www.youtube.com/channel/UC7W_wVS25ecsUs-GOCOQ3QA
- Discord Server - Neckerworld
- Neckerworld was selected to be a participant in Backdrop Build V3 - https://backdropbuild.com/builds/v3/neckerworld
- Citation in "Quantum Mechanics of Human Perception" by Ivan S. Maksymov - https://arxiv.org/abs/2404.10554 pdf: https://arxiv.org/pdf/2404.10554
The following repositories have provided code for this project
- mtserver - https://github.com/vichargrave/threads.git and wqueue and tcpsockets
- rapidjson - THL A29 Limited, a Tencent company, and Milo Yip. All rights reserved.
- stb_image - v2.23 - public domain image loader - http://nothings.org/stb
- twemoji - https://github.com/twitter/twemoji/
And many thanks to collaborator and beta-tester: CS.
This game is still in an early stage of development and many aspects of the server and client are open for enhancement and development. The author is very interested in feedback and comments related to interest and applicability of this work. There is, to the author's knowledge, only one similar effort to build a game where programmed automata are the only players, and that is a racing car game. Future efforts to continue and enhance this game concept will, in part, depend on the feedback received.
- Cube intersection errors -- The display of cubes that are in close proximity sometimes shows unwanted overlap and incorrect physical display.
- The client player (nwplay.py) needs improvement: When cubes run into an obstacle their backup and recovery strategy needs work, better route planning, NoTargets recovery, etc.