The PL-Tokenizer is a summative assessment project for a programming language course. This project demonstrates the implementation of a basic tokenizer that processes and analyzes strings of text. It is designed to categorize and count different types of tokens in the input text, including characters, words, sentences, and special symbols.
The program features a web-based interface where users can input text for analysis. It includes a text area for entering the text and a button to trigger the tokenization process. The results are displayed in an organized manner, showing the total number of characters, words, sentences, and symbols.
- Text Analysis: Breaks down the input text into various token types and counts specific elements such as words, punctuation, and special symbols.
- Dynamic Results: Updates the counts and detailed token breakdown in real-time based on user input.
- Interactive User Interface: Provides a user-friendly experience with automatic adjustment of text area height and clear display of results.
- Tokenization: The program processes the input text to identify and classify tokens (e.g., words, punctuation, special characters).
- Count Display: Shows the total number of characters, words, sentences, and symbols.
- Detailed Breakdown: Provides a detailed output of each token's type and character composition, making it easier to understand the text structure.
- Access the web-based interface.
- Enter the text you want to analyze into the provided input area.
- Click the button to initiate the tokenization process.
- View the summarized and detailed results displayed on the interface.
This project was created by Laurence Lesmoras, Laurence Kharl Devera, and Joshua Famor as part of a programming language course. It serves as an educational tool for understanding basic text processing and tokenization techniques.
Feel free to explore and modify the project as needed. Enjoy experimenting with text analysis!