Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Add CDataProviderIterator to allow iterating over large data sets #546

Merged
merged 9 commits into from

3 participants

Charles Pick Carsten Brandt Alexander Makarov
Charles Pick

Please see #545 for more information

Carsten Brandt cebe commented on the diff
CHANGELOG
@@ -12,6 +12,7 @@ Version 1.1.11 work in progress
- Bug #433: Fixed the bug that Gii model name input autocomplete was not working sometimes (mdomba)
- Bug #454: Removed translation on CDbConnection exception as it was creating an endless loop if the application used CDbCache (mdomba)
- Bug #517: Rule parameter sub-patterns are not checked correctly (ranvis)
+- Enh #545: Add CDataProviderIterator to allow iteration over large data sets
Carsten Brandt Collaborator
cebe added a note

@phpnode your name is missing ;-)

Charles Pick
phpnode added a note

fixed :)

Carsten Brandt Collaborator
cebe added a note

Yeah, thats real time collaboration. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
framework/web/CDataProviderIterator.php
((34 lines not shown))
+ */
+class CDataProviderIterator implements Iterator, Countable
+{
+
+ /**
+ * The data provider to iterate over
+ * @var CDataProvider
+ */
+ public $dataProvider;
+
+ /**
+ * The current index
+ * @var integer
+ */
+ protected $_currentIndex = -1;
+
Carsten Brandt Collaborator
cebe added a note

I think convention is $_ only for private properties. But as you have getters and setters for them, they should be private.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
framework/web/CDataProviderIterator.php
((125 lines not shown))
+ * @return array the items from the next page of results
+ */
+ protected function loadPage()
+ {
+ $this->dataProvider->getPagination()->setCurrentPage($this->_currentPage);
+ return $this->_items = $this->dataProvider->getData(true);
+ }
+ /**
+ * Gets the current item in the list.
+ * This method is required by the Iterator interface
+ * @return mixed the current item in the list
+ */
+ public function current()
+ {
+ $items = $this->getItems();
+ return $items[$this->getCurrentIndex()];
Carsten Brandt Collaborator
cebe added a note

why not return $this->_items[$this->_currentIndex]; ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
framework/web/CDataProviderIterator.php
((149 lines not shown))
+ {
+ $pagination = $this->dataProvider->getPagination();
+ $currentPage = $this->_currentPage;
+ $pageSize = $pagination->getPageSize();
+ return $currentPage * $pageSize + $this->getCurrentIndex();
+ }
+
+ /**
+ * Moves the pointer to the next item in the list.
+ * This method is required by the Iterator interface
+ */
+ public function next()
+ {
+ $pagination = $this->dataProvider->getPagination();
+ $this->_currentIndex++;
+ if ($this->getCurrentIndex() >= $pagination->getPageSize()) {
Carsten Brandt Collaborator
cebe added a note

if ($this->_currentIndex>=$pagination->getPageSize()) { no need to call the getter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
framework/web/CDataProviderIterator.php
((174 lines not shown))
+ */
+ public function rewind()
+ {
+ $this->_currentIndex = 0;
+ $this->_currentPage = 0;
+ $this->loadPage();
+ }
+
+ /**
+ * Checks if the current position is valid or not.
+ * This method is required by the Iterator interface
+ * @return boolean true if this index is valid
+ */
+ public function valid()
+ {
+ return $this->key() < $this->getTotalItems();
Carsten Brandt Collaborator
cebe added a note

$this->_totalItems would do it here too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
framework/web/CDataProviderIterator.php
((184 lines not shown))
+ * This method is required by the Iterator interface
+ * @return boolean true if this index is valid
+ */
+ public function valid()
+ {
+ return $this->key() < $this->getTotalItems();
+ }
+
+ /**
+ * Gets the total number of items in the dataProvider
+ * This method is required by the Countable interface
+ * @return integer the total number of items
+ */
+ public function count()
+ {
+ return $this->getTotalItems();
Carsten Brandt Collaborator
cebe added a note

$this->_totalItems would do it here too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Charles Pick

@cebe thanks for your feedback, implemented your suggestions

Carsten Brandt
Collaborator

Good work, but it is not always the "no spaces" yii coding style.

Charles Pick

@cebe / @qiangxue - is the no spaces style a strict requirement? It would be really good if we had a code sniff for this

Alexander Makarov
Collaborator

It's not a strict requirement but since majority of Yii code is written using it it's better to format pull requests like this. I'm playing with phpcs trying to get a good set of sniffs for Yii core so I'll release config when done.

Carsten Brandt cebe was assigned
Carsten Brandt cebe merged commit b7dff4f into from
Carsten Brandt
Collaborator

Merged and refactored the class in 7150d84 .
Thanks for your work! :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
1  CHANGELOG
View
@@ -12,6 +12,7 @@ Version 1.1.11 work in progress
- Bug #433: Fixed the bug that Gii model name input autocomplete was not working sometimes (mdomba)
- Bug #454: Removed translation on CDbConnection exception as it was creating an endless loop if the application used CDbCache (mdomba)
- Bug #517: Rule parameter sub-patterns are not checked correctly (ranvis)
+- Enh #545: Add CDataProviderIterator to allow iteration over large data sets (phpnode)
Carsten Brandt Collaborator
cebe added a note

@phpnode your name is missing ;-)

Charles Pick
phpnode added a note

fixed :)

Carsten Brandt Collaborator
cebe added a note

Yeah, thats real time collaboration. :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
- Enh #136: Added ability to select database connection in Gii model generator (samdark)
- Enh #165: Allow CCacheDependency to be reusable across multiple cache calls (phpnode)
- Enh #171: Added support for PUT and DELETE request tunneled through POST via parameter named _method in POST body (musterknabe)
217 framework/web/CDataProviderIterator.php
View
@@ -0,0 +1,217 @@
+<?php
+/**
+ * CDataProviderIterator class file.
+ *
+ * @author Charles Pick <charles.pick@gmail.com>
+ * @link http://www.yiiframework.com/
+ * @copyright Copyright &copy; 2008-2012 Yii Software LLC
+ * @license http://www.yiiframework.com/license/
+ */
+
+/**
+ * CDataProviderIterator allows iteration over large data sets without holding the entire set in memory.
+ *
+ * CDataProviderIterator iterates over the results of a data provider, starting at the first page
+ * of results and ending at the last page. It is usually only suited for use with CActiveDataProvider.
+ *
+ * For example, the following code will iterate through all registered users without
+ * running out of memory, even if there are millions of users in the database.
+ * <pre>
+ * $dataProvider = new CActiveDataProvider("User");
+ * $iterator = new CDataProviderIterator($dataProvider);
+ * foreach($iterator as $user) {
+ * echo $user->name."\n";
+ * }
+ * </pre>
+ *
+ * @property array $items the currently loaded items
+ * @property integer $currentIndex the 0 based current index
+ * @property integer $totalItems the total number of items in the iterator
+ *
+ * @author Charles Pick <charles.pick@gmail.com>
+ * @package system.web
+ * @since 1.1.11
+ */
+class CDataProviderIterator implements Iterator, Countable
+{
+
+ /**
+ * The data provider to iterate over
+ *
+ * @var CDataProvider
+ */
+ public $dataProvider;
+
+ /**
+ * The current index
+ *
+ * @var integer
+ */
+ private $_currentIndex=-1;
+
+ /**
+ * The current page in the pagination
+ *
+ * @var integer
+ */
+ private $_currentPage=0;
+
+ /**
+ * The total number of items
+ *
+ * @var integer
+ */
+ private $_totalItems=-1;
+
+ /**
+ * The current set of items
+ *
+ * @var array
+ */
+ private $_items;
+
+ /**
+ * Constructor. Sets the data provider to iterator over
+ *
+ * @param CDataProvider $dataProvider the data provider to iterate over
+ */
+ public function __construct(CDataProvider $dataProvider)
+ {
+ $this->dataProvider=$dataProvider;
+ $this->_totalItems=$dataProvider->getTotalItemCount();
+ }
+
+ /**
+ * Sets the current index position
+ *
+ * @param integer $currentIndex the current index positon
+ */
+ public function setCurrentIndex($currentIndex)
+ {
+ $this->_currentIndex=$currentIndex;
+ }
+
+ /**
+ * Gets the zero based current index position
+ *
+ * @return integer index position
+ */
+ public function getCurrentIndex()
+ {
+ return $this->_currentIndex;
+ }
+
+ /**
+ * Gets the current set of items to iterate over
+ *
+ * @return array the current set items to iterate over
+ */
+ public function getItems()
+ {
+ return $this->_items;
+ }
+
+ /**
+ * Sets the total number of items to iterate over
+ *
+ * @param int $totalItems the total number of items to iterate over
+ */
+ public function setTotalItems($totalItems)
+ {
+ $this->_totalItems=$totalItems;
+ }
+
+ /**
+ * Gets the total number of items to iterate over
+ *
+ * @return integer the total number of items to iterate over
+ */
+ public function getTotalItems()
+ {
+ return $this->_totalItems;
+ }
+
+ /**
+ * Loads the page of results
+ *
+ * @return array the items from the next page of results
+ */
+ protected function loadPage()
+ {
+ $this->dataProvider->getPagination()->setCurrentPage($this->_currentPage);
+ return $this->_items=$this->dataProvider->getData(true);
+ }
+
+ /**
+ * Gets the current item in the list.
+ * This method is required by the Iterator interface
+ *
+ * @return mixed the current item in the list
+ */
+ public function current()
+ {
+ return $this->_items[$this->_currentIndex];
+ }
+
+ /**
+ * Gets the key of the current item.
+ * This method is required by the Iterator interface
+ *
+ * @return integer the key of the current item
+ */
+ public function key()
+ {
+ $pagination=$this->dataProvider->getPagination();
+ $pageSize=$pagination->getPageSize();
+ return $this->_currentPage*$pageSize+$this->_currentIndex;
+ }
+
+ /**
+ * Moves the pointer to the next item in the list.
+ * This method is required by the Iterator interface
+ */
+ public function next()
+ {
+ $pagination=$this->dataProvider->getPagination();
+ $this->_currentIndex++;
+ if($this->_currentIndex>=$pagination->getPageSize())
+ {
+ $this->_currentPage++;
+ $this->_currentIndex=0;
+ $this->loadPage();
+ }
+ }
+
+ /**
+ * Rewinds the iterator to the start of the list.
+ * This method is required by the Iterator interface
+ */
+ public function rewind()
+ {
+ $this->_currentIndex=0;
+ $this->_currentPage=0;
+ $this->loadPage();
+ }
+
+ /**
+ * Checks if the current position is valid or not.
+ * This method is required by the Iterator interface
+ *
+ * @return boolean true if this index is valid
+ */
+ public function valid()
+ {
+ return $this->key()<$this->_totalItems;
+ }
+
+ /**
+ * Gets the total number of items in the dataProvider
+ * This method is required by the Countable interface
+ *
+ * @return integer the total number of items
+ */
+ public function count()
+ {
+ return $this->_totalItems;
+ }
+}
42 tests/framework/web/CDataProviderIteratorTest.php
View
@@ -0,0 +1,42 @@
+<?php
+Yii::import("system.web.*");
+/**
+ * Tests for the {@link CDataProviderIterator} class
+ * @author Charles Pick
+ */
+class CDataProviderIteratorTest extends CTestCase
+{
+ /**
+ * Tests the iterator
+ */
+ public function testIterator()
+ {
+ $dataProvider = new CArrayDataProvider($this->generateData());
+ $iterator = new CDataProviderIterator($dataProvider);
+ $this->assertEquals(100, $iterator->getTotalItems());
+ $this->assertEquals(100, count($iterator));
+ $n = 0;
+ foreach($iterator as $item) {
+ $this->assertEquals("Item ".$n,$item['name']);
+ $n++;
+
+ }
+ }
+
+ /**
+ * Generates some data to fill a dataProvider
+ * @param integer $totalItems the total number of items to generate
+ * @return array the data
+ */
+ protected function generateData($totalItems = 100)
+ {
+ $data = array();
+ for($i = 0; $i < $totalItems; $i++) {
+ $data[] = array(
+ "id" => $i,
+ "name" => "Item ".$i,
+ );
+ }
+ return $data;
+ }
+}
Something went wrong with that request. Please try again.