Skip to content

fix: add encoding parameter to PPI::Document->new#438

Draft
toddr-bot wants to merge 2 commits into
masterfrom
koan.toddr.bot/fix-issue-22
Draft

fix: add encoding parameter to PPI::Document->new#438
toddr-bot wants to merge 2 commits into
masterfrom
koan.toddr.bot/fix-issue-22

Conversation

@toddr-bot

@toddr-bot toddr-bot commented Apr 27, 2026

Copy link
Copy Markdown
Collaborator

Summary

PPI::Document->new($filename) reads files as raw bytes with no encoding support, making it impossible to correctly parse UTF-8 encoded Perl source files containing non-ASCII characters. This adds an encoding parameter so users can specify the file encoding:

my $doc = PPI::Document->new($filename, encoding => 'UTF-8');

Fixes #22

Changes

  • PPI::Util::_slurp() accepts optional encoding parameter, applies :encoding() layer via binmode when specified
  • PPI::Document->new() handles the encoding attribute by slurping with the specified encoding and passing the decoded string to the lexer
  • New PPI::Document->encoding() accessor returns the encoding the document was created with
  • PPI::Document->save() applies the document's encoding when writing, so round-trip load/save preserves the correct byte representation on disk
  • PPI::Document::_setattr() stores the encoding attribute
  • Fully backward compatible: existing code without the encoding parameter is completely unchanged

Test plan

  • New test file t/30_encoding.t with 15 tests covering:
    • Loading UTF-8 files with encoding => 'UTF-8' and verifying character-level (not byte-level) token content
    • PPI::Document::File encoding passthrough
    • Wide characters in string literals
    • Round-trip: load with encoding, save, verify bytes on disk match original
    • encoding accessor method
    • Backward compatibility without encoding parameter
  • Full test suite passes (70 files, 53,106 tests, zero failures)

Generated by Kōan /fix


Quality Report

Changes: 3 files changed, 206 insertions(+), 4 deletions(-)

Code scan: clean

Tests: skipped

Branch hygiene: clean

Generated by Kōan post-mission quality pipeline

toddr-bot and others added 2 commits April 27, 2026 09:37
Add t/30_encoding.t with tests for the new encoding parameter:
- Loading UTF-8 files with encoding => 'UTF-8'
- PPI::Document::File encoding support
- Wide character identifiers in UTF-8 files
- Round-trip preservation of UTF-8 bytes
- encoding accessor method
- Backward compatibility without encoding parameter

Tests are marked with $TODO as the feature is not yet implemented.
Three tests properly fail: character-level length check, wide-char
identifier parsing, and encoding accessor existence.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PPI::Document->new($filename) has always read files as raw bytes with
no encoding layers, making it impossible to correctly parse UTF-8
encoded Perl source files containing non-ASCII characters.

Add an encoding parameter that causes the file to be read with the
specified encoding layer:

  my $doc = PPI::Document->new($file, encoding => 'UTF-8');

Changes:
- PPI::Util::_slurp() accepts optional encoding parameter, applies
  binmode with :encoding() layer when specified
- PPI::Document->new() handles encoding by always using the slurp +
  lex_source path when encoding is given, ensuring the decoded string
  reaches the tokenizer
- PPI::Document->encoding() accessor returns the encoding value
- PPI::Document->save() uses the document's encoding when writing,
  so round-trip load/save preserves the correct byte representation
- PPI::Document::_setattr() stores the encoding attribute
- Remove $TODO markers from t/30_encoding.t tests

Backward compatible: existing code without encoding parameter is
completely unchanged.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

content is read as octets, not characters, with no concept of decoding

2 participants