Contains Lab 1: Makefiles, Linux, And Bash
To get texts from Project Gutenberg, do the following.
-
Navigate to one or more text files (e.g., https://www.gutenberg.org/cache/epub/17192/pg17192.txt).
-
Add text IDs (e.g., 17192) to file Text_IDs.csv.
-
Run command
make get_texts.
To test tokenizer, do the following.
-
Run command
make set_up_virtual_environment_env_upgrade_PIP_and_use_PIP_to_install_Python_packages_specified_in_text_file_requirements. -
Run command
source env/bin/activate. -
Run command
pytest test_tokenizer.py.