Training without subwords

[sebpuetz]

#13 was a first stab at it, maybe we can include this in 0.6?

[danieldk]

See #46 and #47 .

I still don't like the data structures in either PR. I feel like they do not really clearly convey the semantics. I don't have a perfect solution yet. One rough idea is to have a single struct:

struct WordIndex<S>
where
    S: AsRef<[usize]>,
{
    word_idx: usize,
    subword_indices: S,
}

impl WordIndex<[usize; 0]> {
    fn new(word_idx: usize) -> Self {
        WordIndex {
            word_idx,
            subword_indices: Default::default(),
        }
    }
}

impl WordIndex<Vec<usize>> {
    fn new_with_subwords(word_idx: usize, subword_indices: impl Into<Vec<usize>>) -> Self {
        WordIndex {
            word_idx,
            subword_indices: subword_indices.into(),
        }
    }
}

impl<S> WordIndex<S>
where
    S: AsRef<[usize]>,
{
    fn word_index(&self) -> usize {
        *self.word_index
    }

    fn subwords_indices(&self) -> &[usize] {
        self.subword_indices.as_ref()
    }
}

Then the rest of finalfrontier can be oblivious to subwords.

[sebpuetz]

Merged in #54