Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Add generators support #177

Merged
merged 74 commits into from

8 participants

Nikita Popov Xinchen Hui Don't Add Me To Your Organization a.k.a The Travis Bot Kendall Hopkins Stanislav Malyshev Reeze Xia Brad Feehan Account for PHP Pull Requests
Nikita Popov

PR for generators as outlined in https://wiki.php.net/rfc/generators.

(For the diff)

nikic added some commits
Nikita Popov nikic Add T_YIELD "yield" keyword 9b101ac
Nikita Popov nikic Add flag for generator functions
Generator functions have to specify the * (asterix) modifier after the
function keyword. If they do so the ZEND_ACC_GENERATOR flag is added to
the fn_flags.
252f623
Nikita Popov nikic Minor code cleanup
The block for the foreach separator was nested unnecessary. This commit
simply removes that nesting.
9b51a3b
Nikita Popov nikic Add error if yield is used outside a generator
The yield statement can only be used in generator functions, which are
marked with an asterix.
fd2a109
Nikita Popov nikic Add zend_do_suspend_if_generator calls
The execution of generator functions will be suspended right after the
arguments were RECVed. This will be done in zend_do_suspend_if_generator.
e14cfaf
Nikita Popov nikic Add ZEND_SUSPEND_AND_RETURN_GENERATOR opcode
If the function is a generator this opcode will be invoked right after
receiving the function arguments.

The current implementation is just a dummy.
1cec3f1
Nikita Popov nikic Add empty Generator class ca59e54
Nikita Popov nikic Add some boilerplate code for Generator class
The Generator class now uses a zend_generator struct, so it'll be able to
store additional info.

This commit also ensures that Generator cannot be directly instantiated
and extended. The error tests are now in a separate folder from the
(yet-to-come) functional tests.
40b7533
Nikita Popov nikic Make generator functions return a Generator object
Right now generator functions simply immediately return a new Generator
object (no suspension yet).
46fa26a
Nikita Popov nikic Allocate execute_data using malloc for generators
Generators need to switch the execute_data very often. If the execute_data
is allocated on the VM stack this operation would require to always copy
the structure (which is quite large). That's why the execution context is
allocated on the heap instead (only for generators obviously).
5e763d9
Nikita Popov nikic Add initial code for suspending execution
This is just some initial code, which is still quite broken (and needs to be
moved so it can be reused.)
9ce9a7e
Nikita Popov nikic Add dummy Iterator implementation
This simply adds dummy rewind/valid/current/key/next methods to Generator.
2c5ecb4
Nikita Popov nikic Allow calling zend_vm_gen from everywhere
Before one could only call it with cwd=Zend.
ececcbc
Nikita Popov nikic Add support for executing a zend_execute_data
This adds another function execute_ex(), which accepts a zend_execute_data
struct to run (contrary to execute(), which accepts a zend_op_array from
which it initialized the execute_data).

This needs a bit more cleanup.
f627be5
Nikita Popov nikic Add way to pass generator object to opcode handlers
The generator zval is put into the return_value_ptr_ptr.
1a99d1c
Nikita Popov nikic Add YIELD opcode implementation fafce58
Nikita Popov nikic Implement return for generators
For generators ZEND_RETURN directly calls ZEND_VM_RETURN(), thus passing
execution back to the caller (zend_generator_resume).

This commit also adds a check that only return; is used in generators and
not return $value;.
5bb3a99
Nikita Popov nikic Close generator on return d49d397
Nikita Popov nikic Remove wrong dtor call cbfa96c
Nikita Popov nikic Add first real generator test
The test implements an xrange() function (the generator version of range()).
39d3d5e
Nikita Popov nikic Add support for generator methods 247bb73
Nikita Popov nikic Free loop variables
If the generator is closed before it has finished running, it may happen
that some FREE or SWITCH_FREE opcodes haven't been executed and memory is
leaked.

This fixes it by walking the brk_cont_array and manually freeing the
variables.
64a643a
Nikita Popov nikic Fix generator creation when execute_data is not nested
This happens primarily when the generator is invoked from some internal
place like a dynamic function call.
9f52c5c
Nikita Popov nikic Set EG(current_execute_data)
This fixes several issues. In particular it makes method generators work
properly and also allows generators using a symbol table.
bcc7d97
Nikita Popov nikic Properly free resources when generator return value not used
To keep things clean two new functions are introduced:

zend_clean_and_cache_symbol_table(HashTable *symbol_table)
zend_free_compiled_variables(zval ***CVs, int num)
4aab08b
Nikita Popov nikic Make the GOTO and SWITCH VMs work again b770b22
Nikita Popov nikic Add support for $generator->send()
Yield now is an expression and the return value is the value passed to
$generator->send(). By default (i.e. if ->next() is called) the value is
NULL.

Unlike in Python ->send() can be run without priming the generator with a
->next() call first.
3600914
Nikita Popov nikic Allow to use yield without value
If the generator is used as a coroutine it often doesn't make sense to yield
anything. In this case one can simply receive values using

    $value = yield;

The yield here will simply yield NULL.
ad525c2
Nikita Popov nikic Fix segfault when send()ing to a closed generator 12e9283
Nikita Popov nikic Add $generator->close() method
Calling $generator->close() is equivalent to executing a return statement
at the current position in the generator.
72a91d0
Nikita Popov nikic Add support for yielding keys
Keys are yielded using the

    yield $key => $value

syntax. Currently this is implemented as a statement only and not as an
expression, because conflicts arise considering nesting and use in arrays:

    yield yield $a => $b;
    // could be either
    yield (yield $a) => $b;
    // or
    yield (yield $a => $b);

Once I find some way to resolve these conflicts this should be available
as an expression too.

Also the key yielding code is rather copy-and-past-y for the value yielding
code, so that should be factored out.
bc08c2c
Nikita Popov nikic Add auto-increment keys
When no key is explicitely yielded PHP will used auto-incrementing keys
as a fallback. They behave the same as with arrays, i.e. the key is the
successor of the largest previously used integer key.
8790160
Nikita Popov nikic Allow throwing exceptions from generators
The missing piece is how one can find the next stack frame, which is
required for dtor'ing arguments pushed to the stack. As the generator
execute_data does not live on the stack one can't use it to figure out the
start of the next stack frame. So there must be some other method.
0033a52
Nikita Popov nikic Allow yielding during function calls
During function calls arguments are pushed onto the stack. Now these are
backed up on yield and restored on resume. This requires memcpy'ing them,
but there doesn't seem to be any better way to do it.

Also this fixes the issue with exceptions thrown during function calls.
ee89e22
Nikita Popov nikic Make $generator->send() return the current value
This makes the API easier to use (and is consistent with Python and JS).
1477be9
Nikita Popov nikic Add cloning support for generators
Generators can now be cloned. I'm pretty sure that my current code does not
yet cover all the edge cases of cloning the execution context, so there are
probably a few bugs in there :)
6117f4c
Nikita Popov nikic Improve backtraces from generators
The current situation is still not perfect, as the generator function itself
does not appear in the stack trace. This makes sense in some way, but it
would probably be more helpful if it would show up (with the bound arguments)
after the $generator->xyz() call. This could be misleading too though as the
function is not *really* called there.
7b3bfa5
Nikita Popov nikic Properly handle yield during method calls bf82f46
Nikita Popov nikic Fix cloning of generator methods
Forgot to add a reference to the this variable
40760ec
Nikita Popov nikic Fix backtraces and func_get_args()
To make the generator function show up in backtraces one has to insert an
additional execute_data into the chain, as prev_execute_data->function_state
is used to determine the called function.

Adding the additional stack frame is also required for func_get_args(), as
the arguments are fetched from there too. The arguments have to be copied
in order to keep them around. Due to the way they are saved doing so is
quite ugly, so I added another function zend_copy_arguments to zend_execute.c
which handles this.
f169b26
Nikita Popov nikic Add sceleton for yield* expression
This does not yet actually implement any delegation.
d939d2d
Nikita Popov nikic Fix thread safe build 6233408
Nikita Popov nikic Fix segfault in method test
A ref has to be added to $this if the generator is called !nested (which
is the case when it is invoked via getIterator).
1d3f37d
Nikita Popov nikic Implement get_iterator
This implements the get_iterator handler for Generator objects, thus making
direct foreach() iteration significantly faster.
04e781f
Nikita Popov nikic Pass zend_generator directly to Zend VM
Previously the zval* of the generator was passed into the VM by misusing
EG(return_value_ptr_ptr). Now the zend_generator* itself is directly passed
in. This saves us from always having to pass the zval* around everywhere.
14766e1
Nikita Popov nikic Disallow closing a generator during its execution
If a generator is closed while it is running an E_WARNING is thrown and the
call is ignored. Maybe a fatal error should be thrown instead?
ab75ed6
Nikita Popov nikic Forgot to git add two tests 5a9bddb
Nikita Popov nikic Add support by yielding by-reference 85f077c
Nikita Popov nikic Remove asterix modifier (*) for generators
Generators are now automatically detected by the presence of a `yield`
expression in their body.

This removes the ZEND_SUSPEND_AND_RETURN_GENERATOR opcode. Instead
additional checks for ZEND_ACC_GENERATOR are added to the fcall_common
helper and zend_call_function.

This also adds a new function zend_generator_create_zval, which handles
the actual creation of the generator zval from an op array.

I feel like I should deglobalize the zend_create_execute_data_from_op_array
code a bit. It currently changes EG(current_execute_data) and
EG(opline_ptr) which is somewhat confusing (given the name).
c9709bf
Nikita Popov nikic Move a variable 612c249
Nikita Popov nikic Add some more tests 1f70a4c
Nikita Popov nikic Require parenthesis around yield expressions
If yield is used in an expression context parenthesis are now required.
This ensures that the code is unambiguos.

Yield statements can still be used without parenthesis (which should be
the most common case).

Also yield expressions without value can be used without parenthesis,
too (this should be the most common case for coroutines).

If the yield expression is used in a context where parenthesis are required
anyway, no additional parenthesis have to be inserted.

Examples:

    // Statements don't need parenthesis
    yield $foo;
    yield $foo => $bar;

    // Yield without value doesn't need parenthesis either
    $data = yield;

    // Parentheses don't have to be duplicated
    foo(yield $bar);
    if (yield $bar) { ... }

    // But we have to use parentheses here
    $foo = (yield $bar);

This commit also fixes an issue with by-ref passing of $foo[0] like
variables. They previously weren't properly fetched for write.

Additionally this fixes valgrind warnings which were caused by access to
uninitialized memory in zend_is_function_or_method_call().
8074863
Nikita Popov nikic Remove reference restrictions from foreach
foreach only allowed variables to be traversed by reference. This never
really made sense because

    a) Expressions like array(&$a, &$b) can be meaningfully iterated by-ref
    b) Function calls can return by-ref (so they can also be meaningfully
       iterated)
    c) Iterators could at least in theory also be iterated by-ref (not
       sure if any iterator makes use of this)

With by-ref generators the restriction makes even less sense, so I removed
it altogether.
de80e3c
Nikita Popov nikic Fix throwing of exceptions within a generator
If a generator threw an exception and was iterated using foreach (i.e. not
manually) an infinite loop was triggered. The reason was that the exception
was not properly rethrown using zend_throw_exception_internal.
94b2cca
Nikita Popov nikic Throw error also for return occuring before yield
Previously only an error was thrown when return occured after yield. Also
returns before the first yield would fail for by-ref generators.

Now the error message is handled in pass_two, so all returns are checked.
1340893
Nikita Popov nikic Add T_YIELD in tokenizer_data.c
Also had to fix up some tokenizer tests that were affected by the token
number changes.
99f93dd
Nikita Popov nikic Fix implementation of Iterator interface
It looks like you have to implement the Iterator interface *before*
assigning get_iterator. Otherwise the structure for user iterators isn't
correctly zeroed out.

Additionaly I'm setting class_entry->iterator_funcs.funcs now. Not sure if
this is strictly necessary, but better safe than sorry ;)
268740d
Nikita Popov nikic Merge remote-tracking branch 'php-src/master' into addGeneratorsSupport
This is just an intial merge. It does not yet make generators and finally
work together.

Conflicts:
	Zend/zend_language_scanner.c
	Zend/zend_language_scanner_defs.h
	Zend/zend_vm_def.h
	Zend/zend_vm_execute.h
	Zend/zend_vm_execute.skl
	Zend/zend_vm_opcodes.h
f4ce364
Nikita Popov nikic Support trivial finally in generators (no yield, no return)
The finally clause is now properly run when an exception is thrown in the
try-block. It is not yet run on `return` and also not run when the generator
is claused within a try block.

I'll add those two things as soon as laruence refactored the finally code.
ae71693
Nikita Popov nikic Forgot to add test 7195a5b
Nikita Popov nikic Drop Generator::close() method 05f1048
Nikita Popov nikic Fix zts build (typo) 9003cd1
Nikita Popov nikic Merge remote-tracking branch 'php-src/master' into addGeneratorsSupport
Merging master to fix Windows build

Conflicts:
	Zend/zend_language_scanner.c
	Zend/zend_language_scanner_defs.h
	Zend/zend_vm_def.h
1823b16
Nikita Popov nikic Disallow serialization and unserialization f45a0f3
Nikita Popov nikic Merge remote-tracking branch 'php-src/master' into addGeneratorsSupport
Conflicts:
	Zend/zend_vm_def.h
	Zend/zend_vm_execute.h
6517ed0
Nikita Popov nikic Add dedicated opcode for returns from a generator
Generators don't have a return value, so it doesn't make sense to have
a shared implementation here.
68c1e1c
Nikita Popov nikic Finally with return now works in generators too 7cdf636
Nikita Popov nikic Run finally if generator is closed before finishing 4d8edda
Nikita Popov nikic Fix several issues and allow rewind only at/before first yield
 * Trying to resume a generator while it is already running now throws a
   fatal error.
 * Trying to use yield in finally while the generator is being force-closed
   (by GC) throws a fatal error.
 * Rewinding after the first yield now throws an Exception
f53225a
Xinchen Hui

Now I get the diff, thanks

nikic added some commits
Nikita Popov nikic Remove implementation stubs for yield delegation
I decided to leave out yield delegation for an initial proposal, so remove
the stubs for it too.
bd70d15
Nikita Popov nikic Merge remote-tracking branch 'php-src/master' into addGeneratorsSupport
Conflicts:
	Zend/zend_language_parser.y
	Zend/zend_vm_execute.skl
d60e3c6
Stanislav Malyshev smalyshev commented on the diff
Zend/tests/errmsg_043.phpt
@@ -1,12 +0,0 @@
---TEST--
Stanislav Malyshev Owner

why delete this test?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Stanislav Malyshev smalyshev commented on the diff
Zend/zend_compile.c
@@ -6289,9 +6336,7 @@ void zend_do_foreach_cont(znode *foreach_token, const znode *open_brackets_token
if (value->EA & ZEND_PARSED_REFERENCE_VARIABLE) {
assign_by_ref = 1;
- if (!(opline-1)->extended_value) {
Stanislav Malyshev Owner

why this is removed?

Nikita Popov
nikic added a note

I removed the by-ref restrictions for foreach. More info in this commit message: nikic@de80e3c

(Same for the test)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Zend/zend_generators.c
((289 lines not shown))
+ size_t offset = (char *) orig->send_target - (char *) execute_data->Ts;
+ clone->send_target = (temp_variable *) (
+ (char *) clone->execute_data->Ts + offset
+ );
+ Z_ADDREF_P(clone->send_target->var.ptr);
+ }
+
+ if (execute_data->current_this) {
+ Z_ADDREF_P(execute_data->current_this);
+ }
+
+ if (execute_data->object) {
+ Z_ADDREF_P(execute_data->object);
+ }
+
+ /* Prev execute data contains an additional stack frame (for proper)
Stanislav Malyshev Owner

typo - extra ) here

Nikita Popov
nikic added a note

fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reeze Xia reeze commented on the diff
Zend/zend_generators.c
((630 lines not shown))
+ }
+
+ /* The sent value was initialized to NULL, so dtor that */
+ zval_ptr_dtor(&generator->send_target->var.ptr);
+
+ /* Set new sent value */
+ Z_ADDREF_P(value);
+ generator->send_target->var.ptr = value;
+ generator->send_target->var.ptr_ptr = &value;
+
+ zend_generator_resume(generator TSRMLS_CC);
+
+ if (generator->value) {
+ RETURN_ZVAL(generator->value, 1, 0);
+ }
+}
Reeze Xia
reeze added a note

missing a close folder comment /* }}} */ here :)

Nikita Popov
nikic added a note

fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Zend/zend_generators.c
((633 lines not shown))
+ zval_ptr_dtor(&generator->send_target->var.ptr);
+
+ /* Set new sent value */
+ Z_ADDREF_P(value);
+ generator->send_target->var.ptr = value;
+ generator->send_target->var.ptr_ptr = &value;
+
+ zend_generator_resume(generator TSRMLS_CC);
+
+ if (generator->value) {
+ RETURN_ZVAL(generator->value, 1, 0);
+ }
+}
+
+
+/* {{{ proto void Generator::__wakeup
Reeze Xia
reeze added a note

missing a '()'

Nikita Popov
nikic added a note

fixed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
nikic added some commits
Nikita Popov nikic Make sure that exception is thrown on rewind() after closing too cc07038
Nikita Popov nikic Fix segfault when traversing a by-ref generator twice
If you try to traverse an already closed generator an exception will now be
thrown.

Furthermore this changes the error for traversing a by-val generator by-ref
from an E_ERROR to an Exception.
bef7958
Nikita Popov nikic Fix typos dbc7809
Don't Add Me To Your Organization a.k.a The Travis Bot

This pull request fails (merged dbc7809 into 35951d4).

Account for PHP Pull Requests php-pulls merged commit dbc7809 into from
Kendall Hopkins

@nikic I'm getting segfaults with this code https://gist.github.com/3588986

Stanislav Malyshev
Owner

@KendallHopkins please submit a bug report to bugs.php.net so it could be tracked properly.

Brad Feehan bradfeehan commented on the diff
Zend/zend_vm_def.h
@@ -1845,17 +1845,22 @@
zend_bool nested;
zend_op_array *op_array = EX(op_array);
+ /* Generators go throw a different cleanup process */

Through? :smile:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on May 15, 2012
  1. Nikita Popov

    Add T_YIELD "yield" keyword

    nikic authored
Commits on May 19, 2012
  1. Nikita Popov

    Add flag for generator functions

    nikic authored
    Generator functions have to specify the * (asterix) modifier after the
    function keyword. If they do so the ZEND_ACC_GENERATOR flag is added to
    the fn_flags.
  2. Nikita Popov

    Minor code cleanup

    nikic authored
    The block for the foreach separator was nested unnecessary. This commit
    simply removes that nesting.
  3. Nikita Popov

    Add error if yield is used outside a generator

    nikic authored
    The yield statement can only be used in generator functions, which are
    marked with an asterix.
Commits on May 20, 2012
  1. Nikita Popov

    Add zend_do_suspend_if_generator calls

    nikic authored
    The execution of generator functions will be suspended right after the
    arguments were RECVed. This will be done in zend_do_suspend_if_generator.
  2. Nikita Popov

    Add ZEND_SUSPEND_AND_RETURN_GENERATOR opcode

    nikic authored
    If the function is a generator this opcode will be invoked right after
    receiving the function arguments.
    
    The current implementation is just a dummy.
  3. Nikita Popov

    Add empty Generator class

    nikic authored
  4. Nikita Popov

    Add some boilerplate code for Generator class

    nikic authored
    The Generator class now uses a zend_generator struct, so it'll be able to
    store additional info.
    
    This commit also ensures that Generator cannot be directly instantiated
    and extended. The error tests are now in a separate folder from the
    (yet-to-come) functional tests.
  5. Nikita Popov

    Make generator functions return a Generator object

    nikic authored
    Right now generator functions simply immediately return a new Generator
    object (no suspension yet).
Commits on May 23, 2012
  1. Nikita Popov

    Allocate execute_data using malloc for generators

    nikic authored
    Generators need to switch the execute_data very often. If the execute_data
    is allocated on the VM stack this operation would require to always copy
    the structure (which is quite large). That's why the execution context is
    allocated on the heap instead (only for generators obviously).
  2. Nikita Popov

    Add initial code for suspending execution

    nikic authored
    This is just some initial code, which is still quite broken (and needs to be
    moved so it can be reused.)
  3. Nikita Popov

    Add dummy Iterator implementation

    nikic authored
    This simply adds dummy rewind/valid/current/key/next methods to Generator.
Commits on May 24, 2012
  1. Nikita Popov

    Allow calling zend_vm_gen from everywhere

    nikic authored
    Before one could only call it with cwd=Zend.
Commits on May 26, 2012
  1. Nikita Popov

    Add support for executing a zend_execute_data

    nikic authored
    This adds another function execute_ex(), which accepts a zend_execute_data
    struct to run (contrary to execute(), which accepts a zend_op_array from
    which it initialized the execute_data).
    
    This needs a bit more cleanup.
Commits on May 27, 2012
  1. Nikita Popov

    Add way to pass generator object to opcode handlers

    nikic authored
    The generator zval is put into the return_value_ptr_ptr.
  2. Nikita Popov

    Add YIELD opcode implementation

    nikic authored
  3. Nikita Popov

    Implement return for generators

    nikic authored
    For generators ZEND_RETURN directly calls ZEND_VM_RETURN(), thus passing
    execution back to the caller (zend_generator_resume).
    
    This commit also adds a check that only return; is used in generators and
    not return $value;.
  4. Nikita Popov

    Close generator on return

    nikic authored
  5. Nikita Popov

    Remove wrong dtor call

    nikic authored
  6. Nikita Popov

    Add first real generator test

    nikic authored
    The test implements an xrange() function (the generator version of range()).
  7. Nikita Popov
  8. Nikita Popov

    Free loop variables

    nikic authored
    If the generator is closed before it has finished running, it may happen
    that some FREE or SWITCH_FREE opcodes haven't been executed and memory is
    leaked.
    
    This fixes it by walking the brk_cont_array and manually freeing the
    variables.
Commits on May 28, 2012
  1. Nikita Popov

    Fix generator creation when execute_data is not nested

    nikic authored
    This happens primarily when the generator is invoked from some internal
    place like a dynamic function call.
  2. Nikita Popov

    Set EG(current_execute_data)

    nikic authored
    This fixes several issues. In particular it makes method generators work
    properly and also allows generators using a symbol table.
  3. Nikita Popov

    Properly free resources when generator return value not used

    nikic authored
    To keep things clean two new functions are introduced:
    
    zend_clean_and_cache_symbol_table(HashTable *symbol_table)
    zend_free_compiled_variables(zval ***CVs, int num)
Commits on May 29, 2012
  1. Nikita Popov
  2. Nikita Popov

    Add support for $generator->send()

    nikic authored
    Yield now is an expression and the return value is the value passed to
    $generator->send(). By default (i.e. if ->next() is called) the value is
    NULL.
    
    Unlike in Python ->send() can be run without priming the generator with a
    ->next() call first.
  3. Nikita Popov

    Allow to use yield without value

    nikic authored
    If the generator is used as a coroutine it often doesn't make sense to yield
    anything. In this case one can simply receive values using
    
        $value = yield;
    
    The yield here will simply yield NULL.
  4. Nikita Popov
  5. Nikita Popov

    Add $generator->close() method

    nikic authored
    Calling $generator->close() is equivalent to executing a return statement
    at the current position in the generator.
Commits on May 30, 2012
  1. Nikita Popov

    Add support for yielding keys

    nikic authored
    Keys are yielded using the
    
        yield $key => $value
    
    syntax. Currently this is implemented as a statement only and not as an
    expression, because conflicts arise considering nesting and use in arrays:
    
        yield yield $a => $b;
        // could be either
        yield (yield $a) => $b;
        // or
        yield (yield $a => $b);
    
    Once I find some way to resolve these conflicts this should be available
    as an expression too.
    
    Also the key yielding code is rather copy-and-past-y for the value yielding
    code, so that should be factored out.
  2. Nikita Popov

    Add auto-increment keys

    nikic authored
    When no key is explicitely yielded PHP will used auto-incrementing keys
    as a fallback. They behave the same as with arrays, i.e. the key is the
    successor of the largest previously used integer key.
  3. Nikita Popov

    Allow throwing exceptions from generators

    nikic authored
    The missing piece is how one can find the next stack frame, which is
    required for dtor'ing arguments pushed to the stack. As the generator
    execute_data does not live on the stack one can't use it to figure out the
    start of the next stack frame. So there must be some other method.
Commits on May 31, 2012
  1. Nikita Popov

    Allow yielding during function calls

    nikic authored
    During function calls arguments are pushed onto the stack. Now these are
    backed up on yield and restored on resume. This requires memcpy'ing them,
    but there doesn't seem to be any better way to do it.
    
    Also this fixes the issue with exceptions thrown during function calls.
Commits on Jun 1, 2012
  1. Nikita Popov

    Make $generator->send() return the current value

    nikic authored
    This makes the API easier to use (and is consistent with Python and JS).
Commits on Jun 3, 2012
  1. Nikita Popov

    Add cloning support for generators

    nikic authored
    Generators can now be cloned. I'm pretty sure that my current code does not
    yet cover all the edge cases of cloning the execution context, so there are
    probably a few bugs in there :)
  2. Nikita Popov

    Improve backtraces from generators

    nikic authored
    The current situation is still not perfect, as the generator function itself
    does not appear in the stack trace. This makes sense in some way, but it
    would probably be more helpful if it would show up (with the bound arguments)
    after the $generator->xyz() call. This could be misleading too though as the
    function is not *really* called there.
  3. Nikita Popov
Commits on Jun 5, 2012
  1. Nikita Popov

    Fix cloning of generator methods

    nikic authored
    Forgot to add a reference to the this variable
Commits on Jun 9, 2012
  1. Nikita Popov

    Fix backtraces and func_get_args()

    nikic authored
    To make the generator function show up in backtraces one has to insert an
    additional execute_data into the chain, as prev_execute_data->function_state
    is used to determine the called function.
    
    Adding the additional stack frame is also required for func_get_args(), as
    the arguments are fetched from there too. The arguments have to be copied
    in order to keep them around. Due to the way they are saved doing so is
    quite ugly, so I added another function zend_copy_arguments to zend_execute.c
    which handles this.
Commits on Jun 19, 2012
  1. Nikita Popov

    Add sceleton for yield* expression

    nikic authored
    This does not yet actually implement any delegation.
Commits on Jun 21, 2012
  1. Nikita Popov

    Fix thread safe build

    nikic authored
Commits on Jun 23, 2012
  1. Nikita Popov

    Fix segfault in method test

    nikic authored
    A ref has to be added to $this if the generator is called !nested (which
    is the case when it is invoked via getIterator).
  2. Nikita Popov

    Implement get_iterator

    nikic authored
    This implements the get_iterator handler for Generator objects, thus making
    direct foreach() iteration significantly faster.
  3. Nikita Popov

    Pass zend_generator directly to Zend VM

    nikic authored
    Previously the zval* of the generator was passed into the VM by misusing
    EG(return_value_ptr_ptr). Now the zend_generator* itself is directly passed
    in. This saves us from always having to pass the zval* around everywhere.
  4. Nikita Popov

    Disallow closing a generator during its execution

    nikic authored
    If a generator is closed while it is running an E_WARNING is thrown and the
    call is ignored. Maybe a fatal error should be thrown instead?
Commits on Jun 26, 2012
  1. Nikita Popov

    Forgot to git add two tests

    nikic authored
Commits on Jul 17, 2012
  1. Nikita Popov
Commits on Jul 20, 2012
  1. Nikita Popov

    Remove asterix modifier (*) for generators

    nikic authored
    Generators are now automatically detected by the presence of a `yield`
    expression in their body.
    
    This removes the ZEND_SUSPEND_AND_RETURN_GENERATOR opcode. Instead
    additional checks for ZEND_ACC_GENERATOR are added to the fcall_common
    helper and zend_call_function.
    
    This also adds a new function zend_generator_create_zval, which handles
    the actual creation of the generator zval from an op array.
    
    I feel like I should deglobalize the zend_create_execute_data_from_op_array
    code a bit. It currently changes EG(current_execute_data) and
    EG(opline_ptr) which is somewhat confusing (given the name).
  2. Nikita Popov

    Move a variable

    nikic authored
  3. Nikita Popov

    Add some more tests

    nikic authored
Commits on Jul 22, 2012
  1. Nikita Popov

    Require parenthesis around yield expressions

    nikic authored
    If yield is used in an expression context parenthesis are now required.
    This ensures that the code is unambiguos.
    
    Yield statements can still be used without parenthesis (which should be
    the most common case).
    
    Also yield expressions without value can be used without parenthesis,
    too (this should be the most common case for coroutines).
    
    If the yield expression is used in a context where parenthesis are required
    anyway, no additional parenthesis have to be inserted.
    
    Examples:
    
        // Statements don't need parenthesis
        yield $foo;
        yield $foo => $bar;
    
        // Yield without value doesn't need parenthesis either
        $data = yield;
    
        // Parentheses don't have to be duplicated
        foo(yield $bar);
        if (yield $bar) { ... }
    
        // But we have to use parentheses here
        $foo = (yield $bar);
    
    This commit also fixes an issue with by-ref passing of $foo[0] like
    variables. They previously weren't properly fetched for write.
    
    Additionally this fixes valgrind warnings which were caused by access to
    uninitialized memory in zend_is_function_or_method_call().
  2. Nikita Popov

    Remove reference restrictions from foreach

    nikic authored
    foreach only allowed variables to be traversed by reference. This never
    really made sense because
    
        a) Expressions like array(&$a, &$b) can be meaningfully iterated by-ref
        b) Function calls can return by-ref (so they can also be meaningfully
           iterated)
        c) Iterators could at least in theory also be iterated by-ref (not
           sure if any iterator makes use of this)
    
    With by-ref generators the restriction makes even less sense, so I removed
    it altogether.
  3. Nikita Popov

    Fix throwing of exceptions within a generator

    nikic authored
    If a generator threw an exception and was iterated using foreach (i.e. not
    manually) an infinite loop was triggered. The reason was that the exception
    was not properly rethrown using zend_throw_exception_internal.
Commits on Jul 23, 2012
  1. Nikita Popov

    Throw error also for return occuring before yield

    nikic authored
    Previously only an error was thrown when return occured after yield. Also
    returns before the first yield would fail for by-ref generators.
    
    Now the error message is handled in pass_two, so all returns are checked.
  2. Nikita Popov

    Add T_YIELD in tokenizer_data.c

    nikic authored
    Also had to fix up some tokenizer tests that were affected by the token
    number changes.
Commits on Jul 26, 2012
  1. Nikita Popov

    Fix implementation of Iterator interface

    nikic authored
    It looks like you have to implement the Iterator interface *before*
    assigning get_iterator. Otherwise the structure for user iterators isn't
    correctly zeroed out.
    
    Additionaly I'm setting class_entry->iterator_funcs.funcs now. Not sure if
    this is strictly necessary, but better safe than sorry ;)
Commits on Aug 13, 2012
  1. Nikita Popov

    Merge remote-tracking branch 'php-src/master' into addGeneratorsSupport

    nikic authored
    This is just an intial merge. It does not yet make generators and finally
    work together.
    
    Conflicts:
    	Zend/zend_language_scanner.c
    	Zend/zend_language_scanner_defs.h
    	Zend/zend_vm_def.h
    	Zend/zend_vm_execute.h
    	Zend/zend_vm_execute.skl
    	Zend/zend_vm_opcodes.h
  2. Nikita Popov

    Support trivial finally in generators (no yield, no return)

    nikic authored
    The finally clause is now properly run when an exception is thrown in the
    try-block. It is not yet run on `return` and also not run when the generator
    is claused within a try block.
    
    I'll add those two things as soon as laruence refactored the finally code.
Commits on Aug 14, 2012
  1. Nikita Popov

    Forgot to add test

    nikic authored
Commits on Aug 20, 2012
  1. Nikita Popov

    Drop Generator::close() method

    nikic authored
  2. Nikita Popov

    Fix zts build (typo)

    nikic authored
  3. Nikita Popov

    Merge remote-tracking branch 'php-src/master' into addGeneratorsSupport

    nikic authored
    Merging master to fix Windows build
    
    Conflicts:
    	Zend/zend_language_scanner.c
    	Zend/zend_language_scanner_defs.h
    	Zend/zend_vm_def.h
  4. Nikita Popov
Commits on Aug 24, 2012
  1. Nikita Popov

    Merge remote-tracking branch 'php-src/master' into addGeneratorsSupport

    nikic authored
    Conflicts:
    	Zend/zend_vm_def.h
    	Zend/zend_vm_execute.h
  2. Nikita Popov

    Add dedicated opcode for returns from a generator

    nikic authored
    Generators don't have a return value, so it doesn't make sense to have
    a shared implementation here.
  3. Nikita Popov
Commits on Aug 25, 2012
  1. Nikita Popov
  2. Nikita Popov

    Fix several issues and allow rewind only at/before first yield

    nikic authored
     * Trying to resume a generator while it is already running now throws a
       fatal error.
     * Trying to use yield in finally while the generator is being force-closed
       (by GC) throws a fatal error.
     * Rewinding after the first yield now throws an Exception
Commits on Aug 26, 2012
  1. Nikita Popov

    Remove implementation stubs for yield delegation

    nikic authored
    I decided to leave out yield delegation for an initial proposal, so remove
    the stubs for it too.
  2. Nikita Popov

    Merge remote-tracking branch 'php-src/master' into addGeneratorsSupport

    nikic authored
    Conflicts:
    	Zend/zend_language_parser.y
    	Zend/zend_vm_execute.skl
Commits on Aug 30, 2012
  1. Nikita Popov
  2. Nikita Popov

    Fix segfault when traversing a by-ref generator twice

    nikic authored
    If you try to traverse an already closed generator an exception will now be
    thrown.
    
    Furthermore this changes the error for traversing a by-val generator by-ref
    from an E_ERROR to an Exception.
  3. Nikita Popov

    Fix typos

    nikic authored
Something went wrong with that request. Please try again.