This first commit imports files from  mts which was repurposed for bergamot translator
from https://github.com/browsermt/mts/tree/nuke.

Modifications to SentencePiece are necessary to provide token level
string_views. This commit changes marian to an alternate branch which
has the feature incorporated.

CMakeLists have been modified with the necessary includes to add
browsermt/mts@nuke files to the bergamot-translator library. In
addition, adds the ssplit dependency, corresponding includes.

Intel MKL fails on compilation, unable to find libraries. To solve this
3rd_party/CMakeLists.txt is modified with @UG's fixes to propogate
variables (EXT_LIBS, etc) at a library level.

A faster linesplitter added for benchmarks is removed in favour of @UG's
ssplit-cpp.
NOTE: ssplit-cpp's regex based implementation is slow for one-line
parses, which ideally needs to be improved in upstream ssplit-cpp to
trivially reduce to a faster newline character based split.

Commit modifies the example test-code main-mts into the app folder,
updating CMakeLists accordingly.

Removed Alignments, too many questions and no concrete answers. Better
off removing unused code. History is kept for now, for internal use.

Vocabs was earlier loaded in each thread and copied several times.
Modified this to be loaded only once in Service and reference used
consistently later on.

This change makes Tokenizer as a class rather moot, as there's only one
private member and a function. Moved this into TextProcessor.
SentenceSplitter, however remains a separate class.

utils.{h,cpp} had only a single loadVocabularies function, which
is at the moment required only in Service. Making loadVocabularies a
function inside Service and getting rid of utils.*.

- Truncating long sentences into those of a specified length for faster
  processing is now a separate function, for improved readability.
- Changes doing push_back -> emplace_back at places to avoid copy.
- query_to_segments is renamed as process.
- Comments are added in an attempt to bring some sanity.

Only the bergamot-translator library should be linked to main target
Any other library (marian ${MARIAN_CUDA_LIB} ${EXT_LIBS} ssplit
pcrecpp.a pcre.a) should be linked to bergamot-translator target inside
src/translator folder.

Enables Mac and Ubuntu CPU only builds through GitHub CI. CI scripts are
copied from marian-dev with necessary changes.

3rd-party/marian-dev is modified to meet C++17 requirements modifying
for half_float.

Using std::string for config. Now capable of launching marian translator
through API interface. There's a sketchy workaround to convert a string
config to marian::Options, with an added note.

 - Provide yaml formatted string as model configuration
 - Remove redundant files

 - Print original and translated text
 - Just add 2 vector entries for texts

Requirement for string_view is the original source string be transferred
all the way from input to service to back to TranslationResult. This
constraint was violated in several places by means of existence of a
copy-constructor. The issue is fixed by deleting copy and assignment
constructors in marian::bergamot::TranslationResult and
UnifiedAPI::TranslationResult, which demonstrated a few occurances of
the same. Replaced the same with move semantics.  In addition, future is
set and get using move semantics at the moment.  Default
move-constructor didn't seem to be working, so they're made explicit for
TranslationResults.

This commit additionally packs a few deletions and improvements made to
improve structure (textops.cpp, batcher.cpp) along the process of
inspecting and fixing the garbled outputs. They are choose to be kept,
in the interest of time, against a prettified atomic commit engineering.

Combinations of the following commits in jp/string-view-bug
[acfc92 78a588 12d91b 00a277 919e2f 9d3a46 b7e39b 18f67b bf667c]

 - Switch to "wasm" branch of browsermt/marian-dev

 - This change happened because marian submodule changed
   this name

 - Native builds are working fine
   -- bergamot-translator-app output is consistent

 - Added abhi-agg/ssplit-cpp
 - Added its wasm branch in bergamot-translator
 - Native builds of bergamot-translator are successful
   -- Sentence splitting is NOT WORKING
   -- Only translation is working

 - Can be changed back to non-blocking once blocking API
   becomes integrable via WASM port in browser

 - Set cmake option COMPILE_WASM to ON to compile the project
   on WASM

 - Set COMPILE_LIBRARY_ONLY to ON for marian library

 - Make native builds successful with marian decoder
 - COMPILE_DECODER_ONLY flag requires importing some
   compile definitions from marian

  - Make WASM builds successful with marian decoder
  - Setting COMPILE_WASM to ON requires importing some
    compile definitions from marian

 - Using bergamot-translator as a library fails at run time because
   necessary parser options are not set

 - Set COMPILE_THREAD_VARIANT cmake option to ON to compile
   multithreaded variant of the project

 - COMPILE_WASM cmake option sets WASM_BINDINGS compile
   definition that enables code for generating proper JS
   bindings

 - Added "wasm" folder
 - Contains README file as well

 - This fixes the issue of sentencepiece not being able to checkout
   properly

Add  `--recursive` to `git clone` instructions

Update README.md

updating  `--recursive`  on wasm instructions too

…e proper cors headers and wasm mimetype

…ort words per second metric in the log

…cted by the test page

…ault

… model in the test page was faster than elsewhere - the lexical shortlist was not being included at the right place in the model config

 - WORMHOLE cmake option is set to ON when compiling for WASM
 - WASM module might not run on Chrome

This reverts commit 3dd7a60.

Turn of assertions and disable exception catching for wasm builds

 - Includes try/catch free builds
 - Has ASSERTION=0 and DISABLE_EXCEPTION_CATCHING=1 for wasm builds

 - This combination gives min inference time (~ 200 WPS)
   on local machine

…he upcoming release

 - Clears up the spaghetti of model packaging
 - Usage instructions
 - Formatting changes

 - PACKAGE_DIR cmake option can now accept relative paths

 - Now things are consistent with the top level README
   instructions that suggest to build in "build-wasm"
   folder

 - Clarified that the Demo and API usage section assumes
   bergamot models were packaged into wasm binary
 - Formatting changes

…etal builds. Fixes browsermt/bergamot-translator#31

…n circleci

WASM CircleCI builds