nlp.rst

API - Natural Language Processing

Natural Language Processing and Word Representation.

tensorlayer.nlp

generate_skip_gram_batch

sample sample_top

SimpleVocabulary Vocabulary process_sentence create_vocab

simple_read_words read_words read_analogies_file build_vocab build_reverse_dictionary build_words_dataset save_vocab

words_to_word_ids word_ids_to_words

basic_tokenizer create_vocabulary initialize_vocabulary sentence_to_token_ids data_to_token_ids

moses_multi_bleu

Iteration function for training embedding matrix

generate_skip_gram_batch

Sampling functions

Simple sampling

sample

Sampling from top k

sample_top

Vector representations of words

Simple vocabulary class

SimpleVocabulary

Vocabulary class

Vocabulary

Process sentence

process_sentence

Create vocabulary

create_vocab

Read words from file

Simple read file

simple_read_words

Read file

read_words

Read analogy question file

read_analogies_file

Build vocabulary, word dictionary and word tokenization

Build dictionary from word to id

build_vocab

Build dictionary from id to word

build_reverse_dictionary

Build dictionaries for id to word etc

build_words_dataset

Save vocabulary

save_vocab

Convert words to IDs and IDs to words

These functions can be done by Vocabulary class.

List of Words to IDs

words_to_word_ids

List of IDs to Words

word_ids_to_words

Functions for translation

Word Tokenization

basic_tokenizer

Create or read vocabulary

create_vocabulary

initialize_vocabulary

Convert words to IDs and IDs to words

sentence_to_token_ids

data_to_token_ids

Metrics

BLEU

moses_multi_bleu