Handwritten Digits Recognizer using a Perceptron Network
This project is a ready-to-go Python app that lets you learn more about Neural Networks and Machine Learning in general and more about simple Perceptron based handwritten digits recognition in particular.
Try it online
Before installing, you can try it on http://www.sy2002.de/nn.html.
Instead of installing manually you can use Docker, so your system including your Python installation keeps untouched.
Open a terminal window and execute the following command:
git clone https://github.com/sy2002/HWD-Perceptron.git docker run -it -p 5000:5000 sy2002/hwd-perceptron
After your Docker container runs, skip forward to section "Running" for starting the client (HTML) in your web browser.
Python 3.6 (or newer) including
pip and a bash shell on Mac OS,
Linux or Windows. On Windows this might be Cygwin or a
Linux Subsystem (Windows 10+).
Open a terminal window and execute the following commands:
git clone https://github.com/sy2002/HWD-Perceptron.git cd HWD-Perceptron ./install.sh
The following things will happen during the installation:
The Python dependencies (such as numpy, scipy, ...) will be installed.
Data for training the Neural Network will be downloaded from the Internet. Check out server/training/README.md for more details about the training data.
As a visual test, if the training data is valid, the bitmap graphics of three randomly picked digits will be painted to stdout (terminal) as ASCII art.
The Neural Network will be trained and as a result, the trained network will be saved as
Finally, the recognition server is started. Leave the terminal window running and follow the steps in the next section.
Installing inside a Docker container
If you are manually installing inside a Docker container, please make sure
that you run
./install.sh --host=0.0.0.0, instead of
It is also important, that you are adding the port mapping parameter
-p 5000:5000 when invoking Docker, so that the Docker internal port 5000
is mapped to your host and visible from within your host machine's web
Python 2 vs. Python 3 and Virtual Environment
On some systems, there are multiple versions of Python installed. Sometimes
Python 2 can be executed via
python and Python 3 via
project expects, that running
pip yields Python 3. If this
is not the case on your machine, then consider using a python
A Virtual Environment is also useful when there are challenges with the dependencies or collisions with other modules on your system.
Here are the alternate installation instructions that install and run the
whole project in a Virtual Environment. Execute them instead of the
above-mentioned steps. It is assumed that your Python 3 is called
this time (versus
python in the above-mentioned instructions). If your
Python 3 is called otherwise, then just use the appropriate name instead of
python3 in the following steps.
git clone https://github.com/sy2002/HWD-Perceptron.git cd HWD-Perceptron python3 -m venv . source bin/activate ./install.sh
Debian and Ubuntu
On Debian or Ubuntu systems, you may need to install the
package using the following command before you run the above-mentioned
Virtual Environment installation instructions (you may need to use sudo with
apt-get install python3-venv
Manual installation of dependencies
If you run a clean or minimal Ubuntu 16.04 LTS, then you need to install
various dependencies manually, including
Python 3.6 because Ubuntu 10.04 LTS
is installing an older version 3.
manual_dependencies.md shows how to manually install the necessary dependencies on Ubuntu 16.04 LTS.
Even if you use another Linux version, this might be helpful for you to see, what you need to install.
If you used the Docker container or followed the above-mentioned manual installation steps, then you should have the recognition server up and running locally in your terminal window and you should see something like this:
* Serving Flask app "hwdr_server" * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
Hint: In Docker, the IP:Port is shown as 0.0.0.0:5000.
Since the trained network has been saved, you can also run the server after
the successful installation without the need to train it again.
./run_server.sh in a terminal window from the project's
root folder. If you are using a Virtual Environment, then don't forget to
source bin/activate before you run the server.
If you run inside a Docker container that you manually build, please make
sure that you run
In either case, leave the server's terminal window running and open a new
terminal window. Go to the project's root folder and execute
./run_client.sh. Your default web browser should open and show
a page that looks similar to the screenshot above. You can also open this
page manually; it is located at
Now you can start experimenting: Draw a digit and press the recognize button. You should see the recognized number being shown. For the next digit, just start painting over your first digit, the paint area will be cleared automatically, when you paint your next digit. And in the server terminal window, you will see ASCII art version of the digits that you painted.
How it works
Using the MNIST dataset from LeCun et al. has become the "Hello World" example for Neural Networks and basic Machine Learning. Have a look at the folder server/training to learn more about that. The bottom line is, that the network is being trained using 28x28 pixels grayscale images.
The Python server offers one REST method called "recognize", which takes a long string of comma separated floating point values between -1 (black) and 1 (white). These values are representing the 28x28 pixels as a parameter. The method returns the recognized digit between 0 and 9. By default, the above-mentioned installation runs the training script hwdt.py which is initially configured to build a Perceptron network with 28x28 = 784 input nodes, 100 nodes in the hidden layer and 10 output nodes, one for the probability of each recognized digit. The heavy lifting is done inside the flexible Perceptron class spnn.py. Browse the server folder and its README.md to learn more.
Manual installation vs. Docker
In case you installed the project manually, just navigate to the root folder
of the project to start tinkering. In case you used the Virtual Environment,
don't forget to enter
source bin/activate before you start.
In case you used Docker, execute bash when starting the image and navigate to /opt/HWD-Perceptron:
docker run -it -p 5000:5000 sy2002/hwd-perceptron /bin/bash cd /opt/HWD-Perceptron
The manual start of bash as shown here is necessary within the
command because otherwise, the default startup from the
Dockerfile is used, which would immediately start up the server,
preventing you from tinkering.
For your system's web browser (outside Docker) being able to access the
server, you need to run it using
./run_server.sh --host=0.0.0.0 as described
above in the "Running" section.
Inspect the training data
As a first simple experiment: Inspect 25 random digits from the training data and show them as ASCII art on your command line. The script is located at the root folder of the project.
Scroll back your terminal window to see the 25 ASCII arts that might look similar to these examples.
============================= 2 ============================= .--:+- .+#%%%%%%+ +%%%=:::##: =#- .%+ #- *+ #. .* %- #. .. -#: .#%##%- =%+ +%%%%%%%#+-+%+ *%%*++=:+%%%%%- +#= :#%%%=*- +* .:@%#: =#+ %* :-=%%%#. +#= *%%+%%%%#: := :%%%%#=. .-+: ============================= 3 ============================= -*%%@#: %%%%%%%. %%:. :## .. :%= :%+ . -%= -+%#%# +%%%%%. .%%%%%%: .%%#+.## .: :%: =* .%. %= += .=- =+ *%* :%* .%%#=:==+%#. .%%%%%%%%- .*#+--
Manual processing of the client's output
Here is a StackExchange Article that explains how to open the browser's console.
Invoking the REST API manually
nn.htmlin your web browser, for example by using
./run_client.sh. Make sure that your local server is not running as we want to invoke the REST API manually.
Open the browser's console as described above.
Draw a digit and press the Recognize button: The error message "Error connecting to server!" is shown and you should now see a bunch of floating point numbers ranging from -1 to 1 in your console. Leave the browser window including the console open.
Open a new terminal window and start your local server as described above in "Running".
Open a new browser window (leave the other one from step #3 open) and enter this link without pressing enter, yet:
Go back to the window from step #3 and select all the floating point numbers and COPY them.
PASTE the numbers in your new browser window from step #5 by appending them behind the
=sign. Press enter/load the page.
You should see the REST API's response, which is Neuronal Networks's response to the digit, that you have drawn in the above-mentioned step #3.
Running the classification manually as Python script
You need to COPY the floating point number stream from the browser's console, so please execute the steps #1 to #3 and then #6 of the above-mentioned instructions.
Go to the file server/test_cp.py line 29 and comment out this line.
Add a new line directly after line 29:
raw_input = [<PASTE>]
<PASTE>by pasting the number stream you copied in step #1.
Go to the terminal, navigate to the server folder an run
python test_cp.py. You should now see the digit you painted in the browser as ASCII art and you should see the output
Recognized:followed by the correctly recognized digit.
Increasing the recognition accuracy
For saving time during the installation process, only a suboptimally trained
network is being created and saved as
The network has a success rate of 94.87% (failure rate 5.13%). Achieving
better results needs better training, which takes significantly more time than
the training of
Experiment with more optimally trained networks
In the folder server/saved_nn you find networks, that have been previously trained. Have a look at the training results: They show, how different network topologies, learning rates and epochs lead to different recognition qualities.
The above-mentioned training results in the comments section of the training
hwdt.py also correspond to the filenames within
Pick your network. You could for example use
saved_nn/784-2000-10-0.02-epoch-16.npz, which has a recognition rate of
98.14% (failure rate 1.86%). The following steps assume, that you want to use
this network. It is better than our smaller demo network
But as it has 2,000 nodes in its hidden layer, it is consuming more
resources (CPU and RAM) than
Test the network: Open a terminal and go to the server folder:
python load_test.py saved_nn/784-2000-10-0.02-epoch-16.npz training/mnist_test.csv
You should see some basic information about the network, like the topology, the success rate/failure rate and the creation date. Then progress bars are displayed, while the test data is loaded and the test is being performed. Finally, you should see the output should
NETWORK IS OK.
Use the network: Edit line 24 of server/hwdr_server.py to
the_network = "saved_nn/784-2000-10-0.02-epoch-16.npz"
Now run the server (stop any old running instances before) and enjoy the better recognition rate.
Train an own network
Define the network topology that you want by changing line 40 of the training application server/hwdt.py. Here is an example that creates a network, that is a bit "deeper" than the initial demo network:
network_shape = [784, 400, 100, 10]
This topology means: 784 input nodes, then 2 hidden layers with 400 => 100 nodes and then 10 output nodes.
Define the learning rate in line 41. This is a sensitive value that exists between the two extremes: If you choose it too big, then the Gradient Descent inside the Backpropagation algorithm will not find the minimum values of the error functions but bounce around. The result would be bad recognition rates. If you choose it too small, then the training lasts ages. For the sake of the demo you might want to choose
0.03as learning rate (line 41).
Define how many epochs you want the network to be trained. For this example,
3epochs might be a good value (line 42).
Set the prefix of a suitable filename in line 44.
Save the modified
hwdt.pyand run it. Depending on how powerful your machine is, this might take a while. On my Mid 2012 MacBookPro, it took ~12 minutes.
After three epochs, the resulting success rate is 95.11%. This is 0.24 percentage points better than our "quick & dirty" network and it shows, that a more complicated and "deep" network does not necessarily mean "far better results" than the smaller and easier network. You might want to inspire yourself by reading the training results and then play around with topologies, learning rates and epochs.
In the folder
server/saved_nnyou will find one
.npzfile per epoch. You can inspect the success rate of each epoch using
load_test.pyas described above.
Use your network: Go to line 24 of the server server/hwdr_server.py and change it to the filename of your freshly trained network. Stop any running server instance and restart it.