feat: add date time serializion and scientific notation suppression #2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Implemented 8 features to match TypeScript implementation: **Feature 1: DateTime Serialization** - Added datetime and date support to encoder - Convert to ISO 8601 format (YYYY-MM-DD for dates, ISO string for datetimes) - Added type checking in is_primitive() utility **Feature 2: Scientific Notation Suppression** - Added format_float() utility to suppress unnecessary scientific notation - Applied to all float encoding paths (primitive values, arrays, tabular data) - Maintains readability for human-readable ranges **Feature 3: Delimiter Indicators in Array Headers** - Added delimiter indicators: [N\t] for tab, [N|] for pipe - Comma remains default with no indicator - Applied to both root-level and nested tabular arrays - Decoder auto-detects delimiter from header indicator **Feature 4: Smart Delimiter Detection in Decoder** - Decoder checks header indicator first ([N\t] or [N|]) - Falls back to detecting delimiter from first data row - Default to comma if no delimiter detected **Feature 5: Dash Markers in List Arrays** - Added "- " prefix to list array items for visual clarity - Decoder strips dash markers when parsing - Improves readability of non-uniform arrays **Feature 6: Delimiter-Aware Quoting** - Quote strings containing delimiter characters (tab, pipe, comma) - Added delimiter characters to needs_quoting() checks - Prevents parsing ambiguity in tabular data **Feature 7: Indent Auto-Detection** - Added detect_indent() function to analyze TOON strings - Auto-detects 2-space, 4-space, or tab indentation - Supports explicit indent override via options - Threaded indent_size parameter through all parsing functions **Feature 8: Strict Array Length Validation** - Validate declared array count matches actual items in strict mode - Applied to both tabular and list arrays - Raises descriptive ValueError on mismatch - Non-strict mode allows flexible array lengths Test Coverage: - Added 41 new tests (18 encoder, 15 decoder, 8 roundtrip) - All 87 tests passing - Comprehensive coverage of edge cases and error conditions 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR implements 8 features to achieve complete feature parity between the Python and TypeScript TOON library implementations. The changes enable datetime serialization, scientific notation suppression, delimiter indicators in array headers, dash markers in list arrays, smart delimiter detection, indent auto-detection, and strict array validation.
Key changes:
- Enhanced type support with datetime/date serialization to ISO 8601 format
- Improved float formatting to suppress unnecessary scientific notation for better readability
- Added delimiter indicators (
[N\t],[N|]) in array headers with smart detection during decoding - Implemented dash markers (
-) for list array items to improve visual clarity
Reviewed Changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| toon/utils.py | Added datetime/date to primitive types, implemented format_float() for scientific notation suppression |
| toon/encoder.py | Implemented datetime serialization, delimiter indicators in headers, dash markers for list arrays, scientific notation suppression |
| toon/decoder.py | Added detect_indent() function, smart delimiter detection from headers, strict array validation, dash marker parsing |
| tests/test_encoder.py | Added 18 new tests covering datetime encoding, delimiter indicators, dash markers, scientific notation, and root arrays |
| tests/test_decoder.py | Added 15 new tests covering delimiter detection, dash markers, indent detection, strict mode validation |
| tests/test_roundtrip.py | Added 8 new roundtrip tests ensuring encoding/decoding consistency for all new features |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if isinstance(item, float) and (item != item or item == float('inf') or item == float('-inf')): | ||
| encoded_values.append('null') | ||
| if isinstance(item, float): | ||
| if item != item or item == float('inf') or item == float('-inf'): |
There was a problem hiding this comment.
Comparison of identical values; use cmath.isnan() if testing for not-a-number.
| if isinstance(value, float) and (value != value or value == float('inf') or value == float('-inf')): | ||
| return 'null' | ||
| if isinstance(value, float): | ||
| if value != value or value == float('inf') or value == float('-inf'): |
There was a problem hiding this comment.
Comparison of identical values; use cmath.isnan() if testing for not-a-number.
| # Normalize indentation: subtract indent_size spaces to align with first field | ||
| # Original: ' name: extra field' (4 spaces with indent_size=2) | ||
| # Becomes: ' name: extra field' (2 spaces) | ||
| normalized_line = ' ' * expected_indent + next_line.lstrip()[0:] if next_indent >= expected_indent + indent_size else next_line |
There was a problem hiding this comment.
This assignment to 'normalized_line' is unnecessary as it is redefined before this value is used.
This assignment to 'normalized_line' is unnecessary as it is redefined before this value is used.
Suggested change
| normalized_line = ' ' * expected_indent + next_line.lstrip()[0:] if next_indent >= expected_indent + indent_size else next_line |
VinciGit00
changed the title
|
🎉 This PR is included in version 1.3.0 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements 8 features to achieve complete feature parity between the Python and TypeScript versions of the TOON library. All features have been thoroughly tested with 41 new tests added, bringing the total test suite to 87 passing tests.
Features Implemented
1. DateTime Serialization
datetimeanddateobjectsis_primitive()utility to include datetime types2. Scientific Notation Suppression
format_float()utility to suppress unnecessary scientific notation3. Delimiter Indicators in Array Headers
[N\t]for tab,[N|]for pipe4. Smart Delimiter Detection in Decoder
[N\t]or[N|])5. Dash Markers in List Arrays
"- "prefix to list array items for visual clarity6. Delimiter-Aware Quoting
needs_quoting()checks7. Indent Auto-Detection
detect_indent()function to analyze TOON stringsindent_sizeparameter through all parsing functions8. Strict Array Length Validation
ValueErroron mismatchTest Coverage
Files Changed
toon/encoder.py- DateTime support, scientific notation suppression, delimiter indicators, dash markers, key foldingtoon/decoder.py- Smart delimiter detection, indent auto-detection, strict validation, dash marker parsingtoon/utils.py- format_float(), delimiter-aware quoting, updated is_primitive()tests/test_encoder.py- 18 new encoder teststests/test_decoder.py- 15 new decoder teststests/test_roundtrip.py- 8 new roundtrip testsTest Plan
🤖 Generated with Claude Code