This is a "monkey" port of Karpathy' gist "min-char-rnn.py" to swift : https://gist.github.com/karpathy/d4dee566867f8291f086
The network is trained one the sherlock holmes novels to predict the next character.
The rnn has one hidden layer with a simple squash function "tanh". Despite its simplicity the netwok illustrates the elements of making a language model.
The loss starts at 114 and converges to 52. The generated text doesn't make sense because the network and text generator are too simple. Better results would require stacked layers of lstm-neurons and beam search to generate text.
Porting the python version to swift was a very good exercise. The project includes :
- main.swift where you can comment in one of the two swift versions
- rnn_python that uses numpy for alle calculations except for loading the sherlock holmes novels and creation of the dictionary
- rnn_swift that uses TensorFlow as a matrix libray
- package.swift is autogenrated and used to specify dependencies
- simple_rnn.xcodeproj is a project file for running the code in XCcde. The projct file was autogenerated using swift
The convergence of both versions is the same but the memory behaviour is very different:
- rnn_python runs fast but memory isn't release when temporary variable goes out of scope ( even though gc.collect() is called regulary). The network eventually runs out of memory
- rnn_swift runs slowly. Probably because i am running it in debug mode in xcode. Temporary variable are released as they go out of scope and the execution i rock solid !
To set up your environment for ubuntu or macos by following the instructions here : https://github.com/tensorflow/swift/blob/master/Installation.md For macOS do pay attntion to these instructions (I didn't and failed at first):
- set xcode to use Legacy Build System.
- export PATH=/Library/Developer/Toolchains/swift-latest/usr/bin:"${PATH}" if you want to use the swift commandline
Specific for this project: -setup your path in main.swift
If you want to generate your own xcode project then you can do so using :
- "swift package generate-xcodeproj"
- "swift package" to show help instructions