[8.x] Add Builder@lazy() and Builder@lazyById() methods

[JosephSilber]

Background

For querying large datasets, the Builder currently has the cursor() method, which returns a LazyCollection. This uses less memory than a regular Collection (returned from the get() method), since it never keeps more than a single Eloquent model in memory.

However, the cursor() method still has several drawbacks:

1. It's not truly lazy, since PHP still caches all query results in its buffer.

   Turning off buffering introduces its own set of challenges (namely not being able to execute other queries simultaneously).

2. It cannot eager load relationships, since it only ever deals with a single record at a time.

3. Depending on the DB, opening a cursor to a huge dataset may have a slight delay vs. running a query with a LIMIT.

We also have the chunk() method, which is kinda lazy, but with a clunky API — as is evident by the fact that we needed to introduce separate each() and chunkMap() methods (with chunkMap() having to build up the whole result set in memory 🙈).

Introducing lazy()

The new lazy() method introduced in this PR will chunk results behind the scenes, and return a single LazyCollection of results:

$lazyCollection = User::lazy();

Since it's a lazy collection, you have the full power of collections at your fingertips:

You can call each() directly on it:

User::lazy()->each->greet();

You can call map() on it:

$results = User::lazy()->map->calculateOutstandingBalance();

Or even chunk() it... The possibilities are endless, and we'll no longer have to create all of these separate one-off methods to query and manipulate results lazily.

[ejunker]

This looks awesome! Such a powerful feature. I currently have some Builder macros similar to this and they have been critical when working with large data processing tasks. I think it would be great to have them in core so everyone can use them.

[taylorotwell]

Thanks! Would you mind sending me some documentation on this to laravel/docs

[JosephSilber]

Sure! I'll try to whip something up later today or tomorrow.

[tpetry]

Is the cursor() method buffering all results to the memory for every database driver? Until now i expected the method to use real database cursors. I am asking because i got the intention that the postgresql driver seems not to buffer everything to the memory? I should have gotten already some out of memory errors for my laravel postgres applications with large datasets which did not happen. And i really can't find any information whether this is a limitation for mysql or every pdo database.

[stephenjude]

@JosephSilber much love from here. Thanks for this PR

[JosephSilber]

@tpetry The PHP docs only explicitely mention MySQL, so maybe this doesn't apply to other drivers.

This would require testing the memory consumption on different databases, in order to know with any level of certainty.

[decadence]

@JosephSilber can you look at this bug with lazyById please?

Looks like fix will be this check

$lastId = $results->last()->{$alias};

if ($lastId === null) {
    throw new RuntimeException("The lazyById operation was aborted because the [{$alias}] column is not present in the query result.");
}

[decadence]

Fixed here #48436