Merge pull request #19 from vsiivola/update_docs

update docs

.github/workflows/tests.yaml

@@ -11,16 +11,38 @@ jobs:
       matrix:
         include:
           - os: ubuntu-20.04
+            python-version: 3.8
             boost_platform: 20.04
             boost_version: 1.73.0
+          - os: ubuntu-22.04
+            python-version: 3.9
+            boost_platform: 22.04
+            boost_version: 1.81.0
+          - os: ubuntu-22.04
+            python-version: '3.10'
+            boost_platform: 22.04
+            boost_version: 1.81.0
+          - os: ubuntu-22.04
+            python-version: 3.11
+            boost_platform: 22.04
+            boost_version: 1.81.0
           - os: macos-11
+            python-version: 3.11
             boost_platform: 11
             boost_version: 1.73.0
+          - os: macos-12
+            python-version: 3.11
+            boost_platform: 11
+            boost_version: 1.81.0
 
     runs-on: ${{ matrix.os }}
 
     steps:
       - uses: actions/checkout@v3
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python-version }}
       - name: Install boost
         uses: MarkusJx/install-boost@v2.4.1
         id: install-boost

.github/workflows/wheels.yaml

@@ -7,11 +7,11 @@ jobs:
     name: Build wheels on ${{ matrix.os }}
     runs-on: ${{ matrix.os }}
     # build on every tag starting with 'v'
-    # if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
+    if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v')
     strategy:
       fail-fast: false
       matrix:
-        os: [ubuntu-20.04, macos-11]
+        os: [ubuntu-22.04]
 
     steps:
       - uses: actions/checkout@v3

.gitignore

@@ -1,3 +1,5 @@
 *~
 build
-build-*
+build-*
+python-wrapper/varikn.egg-info/
+python-wrapper/varikn/_version.py

INSTALL.md

@@ -0,0 +1,168 @@
+# Installing VariKN
+
+## Table of contents
+
+- [Requirements for source installs](#requirements-for-source-installs)
+  - [How to install MacOsX requirements](#how-to-install-macosx-requirements)
+  - [How to install Linux requirements](#how-to-install-linux-requirements)  
+- [Install Python tools only](#install-python-tools-only)
+  - [Install from PyPI wheel](#install-from-pypi-wheel)
+  - [Install from source](#install-from-source)
+- [Full build](#full-build)
+  - [Build flags and debugging](#build-flags-and-debugging)
+  - [Builds fod IDE toolchains](#builds-fod-ide-toolchains)
+    - [XCode](#xcode)
+    - [Note on Eclipse CDT](#note-on-eclipse-cdt)
+  - [Installing python](#installing-python)
+  - [Testing](#testing)
+
+## Requirements for source installs
+
+If you are on Linux and only need Python tools, you can jump to
+[Install from PYPI wheel](#install-from-pypi-wheel). For other install
+methods, you need a compiler, CMake, and Boost. For building the
+Python tools, also SWIG is required.
+
+### How to install MacOsX requirements
+
+You need a compiler, for example in XCode Command Line
+Tools that can be installed with
+
+```sh
+xcode-select --install
+```
+
+Boost, CMake and SWIG can be installed for with Homebrew with the
+following command.
+
+```sh
+brew install boost cmake swig
+```
+
+### How to install Linux requirements
+
+Check the [Dockerfile](Dockerfile) for an example.
+
+## Install Python tools only
+
+### Install from PyPI wheel
+
+In Linux you should be able to just run the following command to
+install the Python library from [PyPI](https://pypi.org/):
+
+```sh
+python -m pip install varikn
+```
+
+If this for some reason fails, try the next section
+
+### Install from source
+
+You need to have the [required
+tools](#requirements-for-source-installs) to proceed.
+
+```sh
+# Install from the latest source distribution on PyPI:
+python -m pip install varikn --no-binary :all:
+# Install from local source (e.g. cloned Git repository):
+python -m pip install .
+```
+
+## Full build
+
+You need to have the [required
+tools](#requirements-for-source-installs) to proceed.
+
+CMake is a makefile generator (like Autoconf). The build script is in
+CMakeLists.txt.  The convention for CMake is to build "out-of-source",
+where all the work is done in a separate directory from your source
+directory.  First, we configure the build (we only need to do this
+once):
+
+```sh
+mkdir build
+cd build
+cmake .. -DCMAKE_BUILD_TYPE=Release
+cmake --build .
+```
+
+Note that this will build the Python wrapper under `lib/python`, but
+will not install it. See [Install from source](#install-from-source)
+for simple Python installation. You can explicitly disable building
+of the wrapper by passing `-DENABLE_PYTHON_WRAPPER=0` to
+cmake.
+
+### Build flags and debugging
+
+By default, the makefiles generated by CMake are 'quiet'. To make
+them verbose (so you can see what's being passed to gcc, etc) use
+the VERBOSE variable like this: `make VERBOSE=1`
+
+You can make a debug build with
+
+```sh
+cmake .. -DCMAKE_BUILD_TYPE=Debug
+```
+
+If you're experimenting with alternate build flags, you can define
+them as environment variables before configuring.  For example:
+
+```sh
+cd build
+export CXXFLAGS=-fmudflap
+cmake ..
+make
+```
+
+If you are reconfiguring CMake with very different settings (for
+example, switching from Debug to Release), you should start in a
+new build folder or delete everything in your current build folder.
+
+### Builds fod IDE toolchains
+
+CMake also supports generating IDE project files (MSVC, XCode,
+etc). To get a list of what it supports, do `cmake -h`.
+
+#### XCode
+
+For example, to create an MSVC project:
+
+```sh
+mkdir .build
+cd .build
+cmake .. -G "Visual Studio 9 2008"
+start Project.sln
+```
+
+or to create an XCode project:
+
+```sh
+mkdir .build
+cd .build
+cmake .. -G Xcode
+open Project.xcodeproj
+```
+
+#### Note on Eclipse CDT
+
+Eclipse is finicky about the way its projects are laid out.  Instead
+of creating a "build" folder as a child of your source dir you'll need
+to create it as a *sibling*.  You'll also need to tell CMake to link
+the build dir to the source dir in the Eclipse project. (See
+[here](https://gitlab.kitware.com/cmake/community/-/wikis/doc/editors/Eclipse-CDT4-Generator)
+for more info. For example:
+
+```sh
+cd trunk
+mkdir ../.build-trunk
+cd ../.build-trunk
+cmake ../trunk/ -G "Eclipse CDT4 - Unix Makefiles" -DECLIPSE_CDT4_GENERATE_SOURCE_PROJECT=TRUE
+```
+
+Then in Eclipse you'll do File->Import the project in .trunk-build.
+Inside this project you'll see a link to "[Source Directory]".
+### Testing
+
+Building Unit tests can be enabled with `-DENABLE_TESTING=1`. The tests
+do not work on Windows yet. Unit tests can be run with `make test` or
+`ctest --verbose`. Unit tests require the unit test library from Boost.

readme → README.md

@@ -10,8 +10,7 @@ The descriptions of Revised Kneser pruning and Kneser-Ney growing can
 be found in the paper Vesa Siivola, Teemu Hirsimäki and Sami Virpioja,
 "On Growing and Pruning Kneser-Ney Smoothed N-Gram Models", IEEE
 Transactions on Speech, Audio and Language Processing,
-15(5):1617-1624, 2007 (available at
-https://sites.google.com/site/vesassiivola/publications/TASLP2007.pdf).
+15(5):1617-1624, 2007.
 
 The package provides a accurate pruning for Kneser-Ney smoothed
 models. Also, it is possible to train a very high-order n-gram models
@@ -21,17 +20,14 @@ field. You can look at the paper Vesa Siivola, Mathias Creutz and
 Mikko Kurimo: "Morfessor and VariKN machine learning tools for speech
 and language technology", Proceedings of the 8th International
 Conference on Speech Communication and Technology (INTERSPEECH'07),
-2007 (available at
-https://sites.google.com/site/vesassiivola/publications/is2007less.pdf)
-for guidelines on typical use.
+2007 for guidelines on typical use.
 
-For help on installing the toolkit, read the file named "install".
+For help on installing the toolkit, read [INSTALL.md](INSTALL.md).
 
-For help on using the built commands, read the file named "commands.html"
+For help on using the built commands, read [commands.html](commands.html).
 
 If it looks like there is interest in using the toolkit but the
 documentation is inadequate, let me know the main sticking points and
 I'll try to do something about it. If you are interested in a well
 supported toolkit that provides a wide range of functionality, you
-might want to look at the srilm toolkit
-(http://www-speech.sri.com/projects/srilm/).
+might want to look at the [srilm toolkit](http://www.speech.sri.com/projects/srilm/).

docker_notes.md

@@ -0,0 +1,29 @@
+# Rudimentary notes on Docker
+
+```sh
+# Set up docker env (only needed if you use command line docker-machine)
+docker-machine start
+eval $(docker-machine env default)
+
+# Create container
+docker build -t varikn_test_container .
+# Log in container
+docker container run --interactive --tty  varikn_test_container
+# Inside container
+mkdir build; (cd build; cmake .. -DCMAKE_BUILD_TYPE=Release -DENABLE_TESTING=1
+  && make && ctest --verbose)
+
+# clean up
+docker image rm varikn_test_container
+docker machine stop
+```
+
+Notes for mac docker setup
+
+```sh
+# Install needed Homebrew packages
+brew install docker docker-machine
+brew cask install virtualbox
+# Create VM
+docker-machine create default
+```