Skip to content

docs: add CPAN compatibility reports for Scalar::Util and Memoize #497

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
fglock merged 4 commits into from
Apr 13, 2026
Merged

docs: add CPAN compatibility reports for Scalar::Util and Memoize #497

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e11aeae
docs: add CPAN compatibility reports for Scalar::Util and Memoize
fglock Apr 13, 2026
5cd3272
feat: fix isvstring, implement List::Util zip/mesh, bundle Memoize
fglock Apr 13, 2026
e34f411
test: add CPAN test suites for Scalar-List-Utils and Memoize
fglock Apr 13, 2026
3bf3595
docs: mention mesh/zip in bundled-modules.md
fglock Apr 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
172 changes: 172 additions & 0 deletions dev/cpan-reports/Memoize.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,172 @@
# Memoize Compatibility Report for PerlOnJava

> Investigated 2026-04-13 against Memoize 1.17 (CPAN, ARISTOTLE) with PerlOnJava

## Summary

| Metric | Value |
|--------|-------|
| **CPAN distribution** | Memoize-1.17 (ARISTOTLE) |
| **Bundled in PerlOnJava** | No (installed via jcpan for testing) |
| **Test files** | 16 (4 skipped) |
| **Subtests run** | 206 |
| **Subtests explicitly failed** | 0 |
| **Test programs crashed** | 5 / 12 that ran |
| **Test files passing** | 7 / 16 |
| **Overall status** | FAIL (but close to passing for core functionality) |

## Architecture

Memoize is **100% pure Perl** -- no XS required. It caches function return values
by wrapping functions via typeglob manipulation.

Source is available in the Perl 5 checkout at `perl5/cpan/Memoize/` but is
**not bundled** in PerlOnJava's JAR.

## Dependency Analysis

### Direct Dependencies (all satisfied)

| Dependency | Available | Location |
|-----------|-----------|----------|
| `Carp` | Yes | `src/main/perl/lib/Carp.pm` |
| `Scalar::Util` (>=1.11) | Yes (v1.63) | Java backend: `ScalarUtil.java` |
| `Exporter` | Yes | `src/main/perl/lib/Exporter.pm` |
| `warnings` | Yes | Java backend: `Warnings.java` |

### Sub-module Dependencies

| Sub-module | Extra Dependency | Available |
|-----------|------------------|-----------|
| `Memoize::Expire` | `Time::HiRes` | Yes (Java impl) |
| `Memoize::Storable` | `Storable` | Yes (with stub locking) |
| `Memoize::AnyDBM_File` | `AnyDBM_File` | No |
| `Memoize::NDBM_File` | `NDBM_File` | No |
| `Memoize::SDBM_File` | `SDBM_File` | No |

### Perl Language Features Used

| Feature | PerlOnJava Status |
|---------|------------------|
| `*{$name} = $wrapper` (typeglob CODE assign) | Implemented |
| `*{$name}{CODE}` (extract CODE slot) | Implemented |
| `Scalar::Util::set_prototype` | Implemented |
| `caller`, `wantarray` | Implemented |
| `no strict` + symbolic refs | Implemented |
| `prototype()` builtin | Implemented |
| `tied %$hash` | Implemented |
| `warnings::enabled('all')` | Implemented |

## Test Results by File

### Passing (7 files)

| Test File | Subtests | What it tests |
|-----------|----------|---------------|
| t/basic.t | ok | Core memoize/unmemoize, INSTALL, NORMALIZER, prototype preservation |
| t/cache.t | ok | SCALAR_CACHE, LIST_CACHE with MEMORY/FAULT/MERGE |
| t/expmod.t | ok | Memoize::Expire module |
| t/expmod_t.t | ok | Memoize::Expire with timed expiration |
| t/flush.t | ok | flush_cache() |
| t/normalize.t | ok | NORMALIZER option |
| t/unmemoize.t | ok | unmemoize() |

### Failing (5 files)

| Test File | Ran/Planned | Root Cause |
|-----------|-------------|------------|
| t/correctness.t | 16/17 | **StackOverflowError** -- deep recursion test (~100k calls) exceeds JVM stack |
| t/threadsafe.t | 1/8 | `threads` module not available (PerlOnJava limitation) |
| t/tie.t | 0/7 | **StackOverflowError** in `DB_File.pm` line 238/240 (infinite recursion) |
| t/tie_db.t | 0/7 | **StackOverflowError** in `DB_File.pm` (same as tie.t) |
| t/tie_storable.t | 5/6 | 1 subtest not reached (likely `Storable` lock_store stub issue) |

### Skipped (4 files)

| Test File | Reason |
|-----------|--------|
| t/tie_gdbm.t | Could not load `GDBM_File` |
| t/tie_ndbm.t | Could not load `Memoize::NDBM_File` |
| t/tie_odbm.t | Could not load `ODBM_File` |
| t/tie_sdbm.t | Could not load `SDBM_File` |

## Failure Analysis

### 1. StackOverflowError in correctness.t (line 93)

The test probes for the Perl "Deep recursion" warning threshold (~100 recursive calls
in standard Perl) and then verifies that Memoize's wrapper doesn't add extra stack
frames that would trigger the warning. The probe function recurses up to 100,000 times,
which overflows the JVM default stack.

**Workaround**: Run with `JPERL_OPTS="-Xss256m"` to increase JVM stack size.
This is the same workaround used for `re/pat.t` and other recursive tests.

### 2. threads not available (threadsafe.t)

PerlOnJava does not implement Perl-style `threads`. This is a known systemic limitation.
The `CLONE` method in Memoize.pm is defined but harmless.

### 3. DB_File infinite recursion (tie.t, tie_db.t)

`DB_File.pm` has an infinite recursion at lines 238-240. This is a bug in the
`DB_File` shim, not in Memoize itself. These tests tie Memoize's cache to a DB_File
database.

### 4. tie_storable.t partial failure

5 of 6 subtests pass. The final subtest likely involves `lock_store`/`lock_retrieve`
which are stub implementations in PerlOnJava's Storable.

## Core Functionality Assessment

The **core Memoize functionality works correctly**:

- `memoize()` -- caching function return values
- `unmemoize()` -- restoring original functions
- `flush_cache()` -- clearing caches
- `NORMALIZER` -- custom key normalization
- `INSTALL` -- installing under different names
- `SCALAR_CACHE` / `LIST_CACHE` -- cache configuration
- `MERGE` -- merging scalar/list caches
- `Memoize::Expire` -- time-based expiration
- Prototype preservation via `set_prototype`
- Context propagation (`wantarray`)

All failures are in **peripheral features** (threads, DB backends, deep recursion edge case).

## Recommendations

### Bundling Memoize

Memoize is an excellent candidate for bundling. All core dependencies are satisfied.

To bundle, add to `dev/import-perl5/config.yaml`:
```yaml
# Memoize - Function return value caching (pure Perl)
- source: perl5/cpan/Memoize/Memoize.pm
target: src/main/perl/lib/Memoize.pm

- source: perl5/cpan/Memoize/Memoize
target: src/main/perl/lib/Memoize
type: directory
```

### Improving Test Results

1. **correctness.t**: Would pass with `JPERL_OPTS="-Xss256m"` (add to perl_test_runner config)
2. **tie.t / tie_db.t**: Fix DB_File.pm infinite recursion at line 238-240
3. **tie_storable.t**: Investigate the 6th subtest failure
4. **threadsafe.t**: Will always skip/fail (no threads) -- acceptable

### Expected Results After Fixes

| Test | Current | After Fix |
|------|---------|-----------|
| t/correctness.t | FAIL (stack) | PASS (with -Xss256m) |
| t/threadsafe.t | FAIL (threads) | SKIP (acceptable) |
| t/tie.t | FAIL (DB_File) | PASS (after DB_File fix) |
| t/tie_db.t | FAIL (DB_File) | PASS (after DB_File fix) |
| t/tie_storable.t | FAIL (1/6) | Likely PASS |

With these fixes, Memoize would go from 7/16 to 11/16 passing (4 skipped, 1 threads-only).
136 changes: 136 additions & 0 deletions dev/cpan-reports/Scalar-Util.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
# Scalar::Util Compatibility Report for PerlOnJava

> Investigated 2026-04-13 against Scalar-List-Utils 1.70 (CPAN) with PerlOnJava bundled Scalar::Util v1.63

## Summary

| Metric | Value |
|--------|-------|
| **CPAN distribution** | Scalar-List-Utils-1.70 (PEVANS) |
| **Bundled version** | 1.63 (Java backend) |
| **Test files** | 38 |
| **Subtests run** | 816 |
| **Subtests passed** | 606 (74.3%) |
| **Subtests failed** | 210 |
| **Test files passing** | 10 / 38 |
| **Overall status** | FAIL |

## Architecture

Scalar::Util is implemented as a split Perl/Java module:

- **Perl wrapper**: `src/main/perl/lib/Scalar/Util.pm` -- thin shim, uses `XSLoader::load`
- **Java backend**: `src/main/java/org/perlonjava/runtime/perlmodule/ScalarUtil.java` (380 lines)

The CPAN distribution includes **List::Util** and **Sub::Util** alongside Scalar::Util.
Many test failures below are in List::Util or Sub::Util functions, not Scalar::Util itself.

## Function Implementation Status

All 14 standard Scalar::Util EXPORT_OK functions are declared and registered.

| Function | Status | Notes |
|----------|--------|-------|
| `blessed` | Full | Handles blessed refs and `qr//` (implicit "Regexp" blessing) |
| `refaddr` | Full | Uses `System.identityHashCode()` (JVM -- not real memory address) |
| `reftype` | Full | Handles SCALAR, REF, ARRAY, HASH, CODE, GLOB, FORMAT, REGEXP, VSTRING |
| `weaken` | Full | Cooperative reference counting on JVM GC. Well tested. |
| `unweaken` | Full | Restores strong reference |
| `isweak` | Full | Delegates to `WeakRefRegistry.isweak()` |
| `dualvar` | Full | Creates `DualVar` record with separate numeric/string values |
| `isdual` | Full | Checks for DUALVAR type; handles READONLY_SCALAR unwrapping |
| `isvstring` | **Stub** | Always returns false. VSTRING type (ID 5) exists in runtime but is never checked. |
| `looks_like_number` | Full | Delegates to `ScalarUtils.looksLikeNumber()` with fast/slow path |
| `openhandle` | Full | Checks GLOB/GLOBREFERENCE; verifies IO handle not closed; handles `*{}` overload |
| `readonly` | **Partial** | Only detects compile-time constants (`RuntimeScalarReadOnly`). Does NOT detect runtime `Internals::SvREADONLY`. |
| `set_prototype` | Full | Sets/clears prototype on CODE refs |
| `tainted` | **Stub** | Always returns false. Taint mode is not implemented in PerlOnJava. |

## Test Results by File

### Passing (10 files)

| Test File | Subtests | Notes |
|-----------|----------|-------|
| t/any-all.t | ok | List::Util any/all/none/notall |
| t/blessed.t | ok | Scalar::Util blessed |
| t/max.t | ok | List::Util max |
| t/maxstr.t | ok | List::Util maxstr |
| t/minstr.t | ok | List::Util minstr |
| t/prototype.t | ok | Sub::Util set_prototype |
| t/readonly.t | ok | Scalar::Util readonly |
| t/rt-96343.t | ok | Regression test |
| t/stack-corruption.t | ok | Stack safety |
| t/sum0.t | ok | List::Util sum0 |

### Failing (28 files)

| Test File | Ran | Failed | Root Cause |
|-----------|-----|--------|------------|
| t/00version.t | 4 | 1 | Version mismatch: LU::XS reports 1.70 vs bundled LU 1.63 |
| t/dualvar.t | 41 | 3 | `dualvar` increment and UTF-8 handling issues |
| t/exotic_names.t | 238/1560 | 120 | Sub renaming with control characters (`set_subname`); early abort |
| t/first.t | 24 | 6 | `first {}` block not called with `$_` properly |
| t/getmagic-once.t | 6 | 6 | Magic/tie get-magic not invoked correctly |
| t/head-tail.t | 42 | 2 | `head`/`tail` edge cases |
| t/isvstring.t | 3 | 1 | `isvstring` always returns false (stub) |
| t/lln.t | 19 | 1 | `looks_like_number` edge case |
| t/mesh.t | 0/8 | 0 | Crash before any tests run (mesh/zip not implemented) |
| t/min.t | 22 | 1 | `min` edge case |
| t/openhan.t | 21 | 2 | `openhandle` edge cases |
| t/pair.t | 19/29 | 3 | `pairmap`/`pairfirst` issues; early abort |
| t/product.t | 27 | 3 | `product` numeric edge cases |
| t/reduce.t | 33 | 7 | `reduce` block context / prototype issues |
| t/reductions.t | 7 | 1 | `reductions` edge case |
| t/refaddr.t | 32 | 4 | `refaddr` with overloaded/tied objects |
| t/reftype.t | 32 | 3 | `reftype` edge cases (FORMAT, LVALUE) |
| t/sample.t | 9 | 3 | `sample` not implemented or buggy |
| t/scalarutil-proto.t | 12/14 | 1 | Prototype check issues |
| t/shuffle.t | 7 | 1 | `shuffle` edge case |
| t/subname.t | 21 | 7 | `set_subname`/`subname` not fully implemented |
| t/sum.t | 18 | 3 | `sum` numeric edge cases |
| t/tainted.t | 5 | 3 | Taint mode not implemented |
| t/undefined-block.t | 18 | 18 | Undefined code block handling |
| t/uniq.t | 31 | 6 | `uniq`/`uniqstr` edge cases |
| t/uniqnum.t | 23 | 2 | `uniqnum` numeric edge cases |
| t/weak.t | 28 | 2 | Weak reference edge cases |
| t/zip.t | 0/8 | 0 | Crash before any tests run (zip not implemented) |

## Key Failure Categories

### 1. Missing/incomplete List::Util functions
`mesh`, `zip`, `sample` are not implemented or crash. `first`, `reduce`, `reductions` have block-calling issues. These are **List::Util** problems, not Scalar::Util.

### 2. Sub::Util `set_subname`/`subname` (exotic_names.t, subname.t)
Sub renaming with exotic characters (control chars, UTF-8) does not work.
`set_subname` appears non-functional -- renamed closures still report `__ANON__`.

### 3. Stubs returning incorrect values
- `isvstring`: always returns false (trivial fix available)
- `tainted`: always returns false (systemic: no taint mode)

### 4. Magic/tie get-magic (getmagic-once.t)
All 6 tests fail -- get-magic is not invoked the correct number of times.

### 5. Undefined code block handling (undefined-block.t)
All 18 tests fail -- functions don't properly die/warn when passed undefined blocks.

### 6. Version mismatch (00version.t)
Bundled version is 1.63 but CPAN test suite is 1.70. List::Util::XS reports 1.70.

## Existing Test Coverage in PerlOnJava

| Area | Test Files | Coverage |
|------|-----------|----------|
| `weaken`/`isweak`/`unweaken` | 4 files (~634 lines) | Excellent |
| `blessed` | Incidental in subroutine.t | Minimal |
| All other functions | None | No dedicated tests |

## Recommendations

1. **Fix `isvstring`** -- trivial: check `s.type == VSTRING` instead of always returning false
2. **Fix version mismatch** -- update bundled version to 1.70 or sync XS version reporting
3. **Implement `mesh`/`zip`** -- these List::Util functions crash immediately
4. **Fix `first`/`reduce` block calling** -- `$_` not set correctly in the block
5. **Improve `set_subname`** -- critical for Moose/Moo ecosystem
6. **Add unit tests** for untested Scalar::Util functions (blessed, refaddr, reftype, dualvar, etc.)
8 changes: 8 additions & 0 deletions dev/import-perl5/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,14 @@ imports:
- source: perl5/lib/Benchmark.pm
target: src/main/perl/lib/Benchmark.pm

# Memoize - Function return value caching (pure Perl, core since 5.8)
- source: perl5/cpan/Memoize/Memoize.pm
target: src/main/perl/lib/Memoize.pm

- source: perl5/cpan/Memoize/Memoize
target: src/main/perl/lib/Memoize
type: directory

# Net::Ping - Network ping module (required by CPAN::Mirrors)
- source: perl5/dist/Net-Ping/lib/Net/Ping.pm
target: src/main/perl/lib/Net/Ping.pm
Expand Down
Loading
Loading