Skip to content

fix: avoid expanding Ids object#26

Merged
bytestream merged 1 commit intobytestream:masterfrom
ChristophWurst:fix/ids-oom
Apr 23, 2026
Merged

fix: avoid expanding Ids object#26
bytestream merged 1 commit intobytestream:masterfrom
ChristophWurst:fix/ids-oom

Conversation

@ChristophWurst
Copy link
Copy Markdown

I'm hunting an OOM that happens here:

Ids.php:353, Horde_Imap_Client_Ids->_fromSequenceString()
Ids.php:284, Horde_Imap_Client_Ids->_resolveIds()
Ids.php:168, Horde_Imap_Client_Ids->add()
Socket.php:2614, Horde_Imap_Client_Socket->_parseSearch()
Socket.php:2647, Horde_Imap_Client_Socket->_parseEsearch()
Socket.php:4696, Horde_Imap_Client_Socket->_serverResponse()
Socket.php:4597, Horde_Imap_Client_Socket->_getLine()
Socket.php:4321, Horde_Imap_Client_Socket->_sendCmdChunk()
Socket.php:4245, Horde_Imap_Client_Socket->_sendCmd()
Socket.php:2478, Horde_Imap_Client_Socket->_search()
Base.php:2282, Horde_Imap_Client_Base->search()
Base.php:3542, Horde_Imap_Client_Base->resolveIds()
Sync.php:229, Horde_Imap_Client_Data_Sync->__construct()
Base.php:3643, Horde_Imap_Client_Base->sync()

Most likely it's happening because it's a server with QRESYNC but HIGHESTMODSEQ is missing (due to nextcloud/mail#12788 possibly). In any case, having a more efficient representation of ID ranges would be great.

AI-Assisted: Claude Code (Claude Opus 4.6)
Signed-off-by: Christoph Wurst <1374172+ChristophWurst@users.noreply.github.com>
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces a compact interval-based internal representation for Horde_Imap_Client_Ids to prevent OOMs when parsing very large IMAP range strings (notably from ESEARCH/QRESYNC responses).

Changes:

  • Add interval storage ($_intervals) to avoid expanding large start:end ranges into flat arrays.
  • Update add(), remove(), count(), iteration, and string rendering to support interval mode.
  • Add regression and behavior tests for large ranges (merge/remove/serialize/iterate).

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File Description
lib/Horde/Imap/Client/Ids.php Adds interval-based storage and updates core Ids behaviors to operate without expanding large ranges.
test/Horde/Imap/Client/IdsTest.php Adds regression tests to ensure large ranges remain compact and basic operations still work.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +127 to +134
return $last[1];
}
$this->sort();
return end($this->_ids);

case 'min':
if ($this->_intervals !== null) {
return $this->_intervals[0][0];
Copy link

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In interval mode, max assumes $_intervals is non-empty. If an Ids object becomes empty after interval subtraction, end($this->_intervals) returns false and $last[1] will raise a warning/TypeError. Please guard for empty intervals and return the same value as the legacy path on empty (typically false from end([])).

Suggested change
return $last[1];
}
$this->sort();
return end($this->_ids);
case 'min':
if ($this->_intervals !== null) {
return $this->_intervals[0][0];
return ($last === false)
? false
: $last[1];
}
$this->sort();
return end($this->_ids);
case 'min':
if ($this->_intervals !== null) {
return empty($this->_intervals)
? false
: $this->_intervals[0][0];

Copilot uses AI. Check for mistakes.

case 'min':
if ($this->_intervals !== null) {
return $this->_intervals[0][0];
Copy link

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In interval mode, min directly indexes $this->_intervals[0][0]. If the Ids object is empty (e.g., after removing the full range), this will trigger an undefined offset warning. Please add an empty-check and return the same result as the flat-array path on empty (typically false from reset([])).

Suggested change
return $this->_intervals[0][0];
return empty($this->_intervals)
? false
: $this->_intervals[0][0];

Copilot uses AI. Check for mistakes.
Comment on lines 314 to +318
public function reverse()
{
if ($this->_intervals !== null) {
$this->_expandIntervals();
}
Copy link

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reverse() expands intervals into a flat array, which can reintroduce the same OOM this PR is trying to avoid (e.g., _search() can call $sr->reverse() when doing client-side sequence sorting with REVERSE). Consider implementing a reverse/descending iterator mode for interval storage (or otherwise avoiding forced expansion) so very large ranges can still be handled safely.

Copilot uses AI. Check for mistakes.
Comment on lines +416 to +420
$before = memory_get_usage();
$ids->add('1:10000000');
$after = memory_get_usage();
// Flat expansion would need ~400 MB. Intervals use < 1 KB.
$this->assertLessThan(1024 * 1024, $after - $before);
Copy link

Copilot AI Apr 23, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This regression test asserts on memory_get_usage() deltas, which can be flaky across PHP versions/builds/allocators (small allocations can still bump usage by >1MB). A more deterministic assertion would be to inspect internal state via Reflection (e.g., $_intervals non-null / $_ids empty) or to assert behavior that would be impossible with expansion (e.g., strval($ids) stays a compact range without ever touching $ids->ids).

Suggested change
$before = memory_get_usage();
$ids->add('1:10000000');
$after = memory_get_usage();
// Flat expansion would need ~400 MB. Intervals use < 1 KB.
$this->assertLessThan(1024 * 1024, $after - $before);
$ids->add('1:10000000');
$r = new ReflectionClass($ids);
$intervals_prop = $r->getProperty('_intervals');
$intervals_prop->setAccessible(true);
$ids_prop = $r->getProperty('_ids');
$ids_prop->setAccessible(true);
$intervals = $intervals_prop->getValue($ids);
$flat_ids = $ids_prop->getValue($ids);
$this->assertNotNull($intervals);
$this->assertEmpty($flat_ids);

Copilot uses AI. Check for mistakes.
@bytestream bytestream merged commit 0171a72 into bytestream:master Apr 23, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants