Data sets collections #2423

Closed
Ragazzo opened this Issue Feb 14, 2014 · 12 comments

Comments

Projects
None yet
6 participants
@Ragazzo
Contributor

Ragazzo commented Feb 14, 2014

As was noted by @qiangxue when designing Yii2 there was a discussion about implementing collections for AR, but it was decided to make things simple. Current collections examples can be seen for example in L4 Chunking Results | Collections sections and other frameworks. Main use-cases of such collections are:

  • not consuming memory export to pdf/excel and other formats; mainly working with data without involving background jobs and without consuming memory;
  • useful syntax sugar.

I would like to see this feature not only for AR, but mainly abstract with one of implementations for AR though.

@qiangxue

This comment has been minimized.

Show comment
Hide comment
@qiangxue

qiangxue Feb 14, 2014

Member

@Ragazzo Would you be able to provide some code snippets to explain the use cases? Also note that #2409 already attempts to solve the memory problem when working with large set of query results (not just AR) using Query and ActiveQuery.

Member

qiangxue commented Feb 14, 2014

@Ragazzo Would you be able to provide some code snippets to explain the use cases? Also note that #2409 already attempts to solve the memory problem when working with large set of query results (not just AR) using Query and ActiveQuery.

@nineinchnick

This comment has been minimized.

Show comment
Hide comment
@nineinchnick

nineinchnick Feb 14, 2014

Contributor

I'll provide some later.

Contributor

nineinchnick commented Feb 14, 2014

I'll provide some later.

@Ragazzo

This comment has been minimized.

Show comment
Hide comment
@Ragazzo

Ragazzo Feb 14, 2014

Contributor

Would you be able to provide some code snippets to explain the use cases?

yes, but as i said above currently as i see it is for easy export, to avoid background jobs. However i find it useful as sytax sugar, that i miss from ruby iterators/collections for example, where it is easy to filter and use them on the fly). I am not forcing it though.

Contributor

Ragazzo commented Feb 14, 2014

Would you be able to provide some code snippets to explain the use cases?

yes, but as i said above currently as i see it is for easy export, to avoid background jobs. However i find it useful as sytax sugar, that i miss from ruby iterators/collections for example, where it is easy to filter and use them on the fly). I am not forcing it though.

@Ragazzo

This comment has been minimized.

Show comment
Hide comment
@Ragazzo

Ragazzo Feb 14, 2014

Contributor

Also note that #2409 already attempts to solve the memory problem when working with large set of query results (not just AR) using Query and ActiveQuery.

yeah i know this, in L4 it also not in collection and in AR. However this is a good syntax sugar.

Contributor

Ragazzo commented Feb 14, 2014

Also note that #2409 already attempts to solve the memory problem when working with large set of query results (not just AR) using Query and ActiveQuery.

yeah i know this, in L4 it also not in collection and in AR. However this is a good syntax sugar.

@samdark

This comment has been minimized.

Show comment
Hide comment
@samdark

samdark Feb 14, 2014

Member

Export at collection level sounds weird. What if I need to add my own format? What if I need to customize JSON I want to get from my data (i.e. use just some fields)? To me a separate converter that takes input and returns output is much better.

Member

samdark commented Feb 14, 2014

Export at collection level sounds weird. What if I need to add my own format? What if I need to customize JSON I want to get from my data (i.e. use just some fields)? To me a separate converter that takes input and returns output is much better.

@Ragazzo

This comment has been minimized.

Show comment
Hide comment
@Ragazzo

Ragazzo Feb 14, 2014

Contributor

Export at collection level sounds weird

i mean not implementing this methods in collection, but fetching chunked result, so it is more of #2409 issue and chunk feature. So currently it is good syntax sugar + collections/iterators feature that is not native for php but native for ruby.

Contributor

Ragazzo commented Feb 14, 2014

Export at collection level sounds weird

i mean not implementing this methods in collection, but fetching chunked result, so it is more of #2409 issue and chunk feature. So currently it is good syntax sugar + collections/iterators feature that is not native for php but native for ruby.

@samdark

This comment has been minimized.

Show comment
Hide comment
@samdark

samdark Feb 14, 2014

Member

Yeah, #2409 solves memory problem perfectly. So that leaves us with syntax sugar/iterators.

Member

samdark commented Feb 14, 2014

Yeah, #2409 solves memory problem perfectly. So that leaves us with syntax sugar/iterators.

@samdark

This comment has been minimized.

Show comment
Hide comment
@samdark

samdark Feb 14, 2014

Member

I've briefly reviewed L4 guide you've mentioned. It seems what they are wrapping multiple ARs with is very similar to Yii 1.1 collections such as CMap or CList that were basically array-like structures with some OO-syntax and that we've dropped in 2.0 since we've realised it's not any better than plain arrays. And less performant btw.

Syntaxwise collection gives you nothing out of the box. Let's compare to arrays:

// checking for a key
if ($collection->contains(2)) {
if (isset($array[2]) {

// converting to array, in case of array it's not necessary
$array = $collection->toArray();

// converting to json
$json = $collection->toJson();
$json = Json::encode($array);

// iterating
$collection = $collection->each(function($item) {
foreach($array as $item) {

// filtering
$collection = $collection->filter(function($item) {
$array = array_filter($array, function($item) {

// applying callback to each object
$collection->each(function($item) { ... });
$array = array_map(function($item) { ... }, $array);

// sorting by value
$collection = $collection->sortBy('created_at');
ArrayHelper::multisort($array, 'created_at');

The only thing is that there's a possibility to define newCollection method per model to return a custom collection but that's what we need use cases for.

Member

samdark commented Feb 14, 2014

I've briefly reviewed L4 guide you've mentioned. It seems what they are wrapping multiple ARs with is very similar to Yii 1.1 collections such as CMap or CList that were basically array-like structures with some OO-syntax and that we've dropped in 2.0 since we've realised it's not any better than plain arrays. And less performant btw.

Syntaxwise collection gives you nothing out of the box. Let's compare to arrays:

// checking for a key
if ($collection->contains(2)) {
if (isset($array[2]) {

// converting to array, in case of array it's not necessary
$array = $collection->toArray();

// converting to json
$json = $collection->toJson();
$json = Json::encode($array);

// iterating
$collection = $collection->each(function($item) {
foreach($array as $item) {

// filtering
$collection = $collection->filter(function($item) {
$array = array_filter($array, function($item) {

// applying callback to each object
$collection->each(function($item) { ... });
$array = array_map(function($item) { ... }, $array);

// sorting by value
$collection = $collection->sortBy('created_at');
ArrayHelper::multisort($array, 'created_at');

The only thing is that there's a possibility to define newCollection method per model to return a custom collection but that's what we need use cases for.

@grikdotnet

This comment has been minimized.

Show comment
Hide comment
@grikdotnet

grikdotnet Feb 14, 2014

The taste of sugar comes when you folow the MVC pattern more precisely, avoiding data processing in controller as much as possible, and when you have domain layer models separated from the table-mapping autogenerated models.
In this case you enjoy passing query results wrapped by a collection to a domain layer model with a type hint in the setter method.
You can incapsulate processing of the empty results, forcing other developers to take care by throwing an exception from your collection, or by making it a null-object that can be referenced safely regardless of the query result.
80% of developers won't even understand what it is all about cause it's always easier to switch off notices, but sometimes ... you may SUDDENLY need a slave server and have to rewrite a whole CDBconnection to support replication, when you thought 640 kb will be enough for everyone :)

The taste of sugar comes when you folow the MVC pattern more precisely, avoiding data processing in controller as much as possible, and when you have domain layer models separated from the table-mapping autogenerated models.
In this case you enjoy passing query results wrapped by a collection to a domain layer model with a type hint in the setter method.
You can incapsulate processing of the empty results, forcing other developers to take care by throwing an exception from your collection, or by making it a null-object that can be referenced safely regardless of the query result.
80% of developers won't even understand what it is all about cause it's always easier to switch off notices, but sometimes ... you may SUDDENLY need a slave server and have to rewrite a whole CDBconnection to support replication, when you thought 640 kb will be enough for everyone :)

@Ragazzo

This comment has been minimized.

Show comment
Hide comment
@Ragazzo

Ragazzo Feb 14, 2014

Contributor

The taste of sugar comes when you folow the MVC pattern more precisely, avoiding data processing in controller as much as possible, and when you have domain layer models separated from the table-mapping autogenerated models.

could be, but current issue is not about MVC at all, it is only about iterators/collections that are native in ruby for example and not in php. However i doubt this will be implemented in fw, because php has its own way and vision for historical reasons.

You can incapsulate processing of the empty results, forcing other developers to take care by throwing an exception from your collection, or by making it a null-object that can be referenced safely regardless of the query result.

true, but it is like making too complicated design that will not be needed, i also find it not useful to use null-objects, since they are like some pointers that your system has too much OOP.

80% of developers won't even understand what it is all about

depends on what language they were programming, if for ruby it is very native there and everyone knows what is it for)


Anyway as @samdark pointed that since this idea was not adopted by many developers in Yii1 i doubt it worth implementing in Yii2 after all. batch feature is good though.

Contributor

Ragazzo commented Feb 14, 2014

The taste of sugar comes when you folow the MVC pattern more precisely, avoiding data processing in controller as much as possible, and when you have domain layer models separated from the table-mapping autogenerated models.

could be, but current issue is not about MVC at all, it is only about iterators/collections that are native in ruby for example and not in php. However i doubt this will be implemented in fw, because php has its own way and vision for historical reasons.

You can incapsulate processing of the empty results, forcing other developers to take care by throwing an exception from your collection, or by making it a null-object that can be referenced safely regardless of the query result.

true, but it is like making too complicated design that will not be needed, i also find it not useful to use null-objects, since they are like some pointers that your system has too much OOP.

80% of developers won't even understand what it is all about

depends on what language they were programming, if for ruby it is very native there and everyone knows what is it for)


Anyway as @samdark pointed that since this idea was not adopted by many developers in Yii1 i doubt it worth implementing in Yii2 after all. batch feature is good though.

@grikdotnet

This comment has been minimized.

Show comment
Hide comment
@grikdotnet

grikdotnet Feb 14, 2014

It would be useful to be able to point a collection class to wrap around a result set, or even just make an afterFindAll callback, cause it is easy to avoid collections in all cases. Vice versa, CMap and CList were slow and ugly.
Sinse the memory issue is solved with delayed instantiating, collections in results are optional. I offered collections 2 years ago as a way to solve memory consumption and keep oop-style, but most coders prefer plain arrays anyway.

It would be useful to be able to point a collection class to wrap around a result set, or even just make an afterFindAll callback, cause it is easy to avoid collections in all cases. Vice versa, CMap and CList were slow and ugly.
Sinse the memory issue is solved with delayed instantiating, collections in results are optional. I offered collections 2 years ago as a way to solve memory consumption and keep oop-style, but most coders prefer plain arrays anyway.

@qiangxue qiangxue added this to the 2.1 milestone Apr 16, 2014

@Ragazzo Ragazzo closed this Jun 27, 2014

@cebe cebe removed this from the 2.1 milestone Jul 23, 2014

@cebe

This comment has been minimized.

Show comment
Hide comment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment