Overview
A multi-language lexical analyzer that tokenizes source code from Python, C, C++, and Java using regular expressions. This tool parses code into categorized tokens (keywords, identifiers, literals, comments, etc.) and provides statistical analysis of token usage.

Supported Languages
Python (.py)
C (.c, .h)
C++ (.cpp, .cc, .hpp)
Java (.java)

Features
Accurate token recognition for 4 programming languages
Handles complex syntax elements:
Multi-line comments and docstrings
Raw strings and special literals
Preprocessor directives (#include, #define)
Annotations and decorators
Template/generic syntax
Lambda expressions
File extension-based language detection
Interactive and file-based input modes
Token classification and counting statistics

Installation & Requirements
Python 3.x
No external dependencies (uses only standard library modules: re, os)

Follow the prompts:
Choose input mode (manual entry or file)
For files, provide path (language auto-detected)
For manual mode, specify language and paste code
View token output and statistics

Token Types Recognized
The tokenizer identifies over 20 different token types including:

KEYWORDS (language reserved words)
IDENTIFIERS (variable/function names)
TYPES (data types)
OPERATORS (+, -, *, /, etc.)
LITERALS (strings, numbers, booleans)
COMMENTS (single-line, multi-line, documentation)
DIRECTIVES (preprocessor commands)
SYMBOLS (braces, parentheses, punctuation)