Repository navigation
VirtLang v3.4.0
VirtLang v3.4.0 Release Notes: Enhanced String Literal Processing with Escape Sequence Support
We are pleased to announce the release of VirtLang v3.4.0! This version significantly enhances the capabilities of string literals within the VirtLang language by introducing comprehensive support for escape sequences.
This much-anticipated update allows developers to embed special characters like newlines, tabs, carriage returns, and even Unicode characters directly within their string literals, making string manipulation more intuitive and powerful.
Major Features & Enhancements
- Full Escape Sequence Support in String Literals (Lexer):
-
Benefit: Developers can now define strings containing special characters (e.g., newlines, tabs, quotes) using standard escape sequences, leading to more readable and expressive code.
-
Details: The VirtLang-Go lexer has been upgraded to correctly parse and unescape the following sequences within string literals (both single-quoted
'...'and double-quoted"..."):\n: Newline\t: Tab\r: Carriage Return\b: Backspace (Note: actual behavior in output might depend on terminal/display)\f: Form Feed (Note: actual behavior in output might depend on terminal/display)\\: Backslash\": Double Quote (within a double-quoted string)\': Single Quote (within a single-quoted string)\uXXXX: Unicode character specified by 4 hexadecimal digits (e.g.,"\u2388"for Helm symbol Helm symbol).- (Note: Octal escapes like
\033are not currently part of this update but may be considered in the future.)
-
Implementation:
- A new internal helper function
lexer.UnescapeString(s string)has been introduced, utilizingstrconv.Unquotefor robust unescaping. - The lexer's string tokenization logic now correctly identifies and processes these escape sequences, ensuring the
Token.Literalfor string tokens contains the actual unescaped characters (while still being wrapped in its original quotes for consistency if it was originally quoted). - Multi-line strings are also correctly handled, with escape sequences processed per line before rejoining.
- A new internal helper function
-
Example:
let greeting = "Hello,\n\tWorld! \"Welcome\" to VirtLang \u2728."; // The 'greeting' variable will now actually contain a newline, a tab, // an embedded double quote, and the sparkles Unicode character. let singleQuotes = 'It\'s a \u263A day!'; // 'singleQuotes' will contain an embedded single quote and a smiley face.
-
Core Lexer Enhancements
- String Tokenization Logic (
lexer/lexer.go):- The loop for consuming string content now correctly handles escaped quote characters (
\",\') to prevent premature termination of the string token. - After collecting the raw string content (between the opening and closing quotes), it now processes this content line-by-line (handling
\r\nand\ras\n), unescaping each line using the newUnescapeStringhelper before constructing the final token literal. This ensures correct unescaping even in multi-line raw strings. - Improved error reporting for invalid escape sequences within strings, leveraging errors from
strconv.Unquote.
- The loop for consuming string content now correctly handles escaped quote characters (
Testing
- New Lexer Tests (
lexer/lexer_test.go):- Added a dedicated test suite
TestUnescapeStringto validate the behavior of the newlexer.UnescapeStringhelper function with various inputs, including simple strings, direct unquotes, multiple escape sequences, empty strings, and invalid sequences. - Expanded
TestTokenizewith new test cases specifically covering:- Basic escape sequences (
\n,\t,\r). - Strings with multiple mixed escape sequences.
- Strings containing all supported escape sequences (
\n,\t,\r,\b,\f,\\,\",\'). - Unicode escape sequences (
\uXXXX). - Previously problematic cases like strings containing literal newlines (CRLF) are now also correctly tokenized with their content unescaped (newlines preserved).
- Basic escape sequences (
- Added a dedicated test suite
Known Issues and Limitations
- Octal Escapes: Standard octal escape sequences (e.g.,
\033) are not currently supported. - Backspace/Formfeed Display: The actual rendering of
\b(backspace) and\f(form feed) is terminal/environment dependent and might not always produce a visible effect as expected in all output contexts.
Upgrade Notes
- This release significantly changes how string literals are processed by the lexer.
- Potential Behavior Change: If your existing VirtLang-Go code contained string literals with character sequences that happen to look like escape sequences (e.g., a literal
\followed by annlike"\\n") but were intended to be literal backslashes and 'n's, these will now be interpreted as actual newline characters.- To represent a literal backslash, you must now use
\\. For example, to get the string\n, you would write"\\n".
- To represent a literal backslash, you must now use
- It is highly recommended to review any string literals in your existing code that use backslashes to ensure they behave as intended with the new unescaping logic.
- This change makes VirtLang-Go strings more aligned with conventions in many other programming languages.
Getting Started
- Update to VirtLang-Go v3.4.0.
- You can now freely use standard escape sequences in your string literals.
Acknowledgements
A significant thank you to @dev-kas for implementing this crucial enhancement to string handling in VirtLang-Go, greatly improving its expressiveness and usability!