New Component: Expression Language #8913

Merged
merged 12 commits into from Sep 19, 2013

Projects

None yet
@fabpot
Symfony member
Q A
Bug fix? no
New feature? yes
BC breaks? no
Deprecations? no
Tests pass? yes
Fixed tickets #8850, #7352
License MIT
Doc PR not yet

TODO:

  • write documentation
  • add tests for the new component
  • implement expression support for access rules in the security component
  • find a better character/convention for expressions in the YAML format
  • check the performance of the evaluation mode
  • better error messages in the evaluation mode
  • add support in the Routing
  • add support in the Validator

The ExpressionLanguage component provides an engine that can compile and
evaluate expressions.

An expression is a one-liner that returns a value (mostly, but not limited to, Booleans).

It is a strip-down version of Twig (only the expression part of it is
implemented.) Like Twig, the expression is lexed, parsed, and
compiled/evaluated. So, it is immune to external injections by design.

If we compare it to Twig, here are the main big differences:

  • only support for Twig expressions
  • no ambiguity for calls (foo.bar is only valid for properties, foo['bar'] is only valid for array calls, and foo.bar() is required for method calls)
  • no support for naming conventions in method calls (if the method is named getFoo(), you must use getFoo() and not foo())
  • no notion of a line for errors, but a cursor (we are mostly talking about one-liners here)
  • removed everything specific to the templating engine (like output escaping or filters)
  • no support for named arguments in method calls
  • only one extension point with functions (no possibility to define new operators, ...)
  • and probably even more I don't remember right now
  • there is no need for a runtime environment, the compiled PHP string is self-sufficient

An open question is whether we keep the difference betweens arrays and hashes.

The other big difference with Twig is that it can work in two modes (possible
because of the restrictions described above):

  • compilation: the expression is compiled to PHP and is self-sufficient
  • evaluation: the expression is evaluated without being compiled to PHP (the node tree produced by the parser can be serialized and evaluated afterwards -- so it can be saved on disk or in a database to speed up things when needed)

Let's see a simple example:

$language = new ExpressionLanguage();

echo $language->evaluate('1 + 1');
// will echo 2

echo $language->compile('1 + 2');
// will echo "(1 + 2)"

The language supports:

  • all basic math operators (with precedence rules):

    • unary: not, !, -, +
    • binary: or, ||, and, &&, b-or, b-xor, b-and, ==, ===, !=, !==, <, >, >=, <=, not in, in, .., +, -, ~, , /, %, *
  • all literals supported by Twig: strings, numbers, arrays ([1, 2]), hashes
    ({a: "b"}), Booleans, and null.

  • simple variables (foo), array accesses (foo[1]), property accesses
    (foo.bar), and method calls (foo.bar(1, 2)).

  • the ternary operator: true ? true : false (and all the shortcuts
    implemented in Twig).

  • function calls (constant('FOO') -- constant is the only built-in
    functions).

  • and of course, any combination of the above.

The compilation is better for performances as the end result is just a plain PHP string without any runtime. For the evaluation, we need to tokenize, parse, and evaluate the nodes on the fly. This can be optimized by using a ParsedExpression or a SerializedParsedExpression instead:

$nodes = $language->parse($expr, $names);
$expression = new SerializedParsedExpression($expr, serialize($nodes));

// You can now store the expression in a DB for later reuse

// a SerializedParsedExpression can be evaluated like any other expressions,
// but under the hood, the lexer and the parser won't be used at all, so it''s much faster.
$language->evaluate($expression);

That's all folks!

I can see many use cases for this new component, and we have two use cases in
Symfony that we can implement right away.

Using Expressions in the Service Container

The first one is expression support in the service container (it would replace
#8850) -- anywhere you can pass an argument in the service container, you can
use an expression:

$c->register('foo', 'Foo')->addArgument(new Expression('bar.getvalue()'));

You have access to the service container via this:

container.get("bar").getvalue(container.getParameter("value"))

The implementation comes with two functions that simplifies expressions
(service() to get a service, and parameter to get a parameter value). The
previous example can be simplified to:

service("bar").getvalue(parameter("value"))

Here is how to use it in XML:

<parameters>
    <parameter key="value">foobar</parameter>
</parameters>
<services>
    <service id="foo" class="Foo">
        <argument type="expression">service('bar').getvalue(parameter('value'))</argument>
    </service>
    <service id="bar" class="Bar" />
</services>

and in YAML (I chose the syntax randomly ;)):

parameters:
    value: foobar

services:
    bar:
        class: Bar

    foo:
        class: Foo
        arguments: [@=service("bar").getvalue(parameter("value"))]

When using the container builder, Symfony uses the evaluator, but with the PHP
dumper, the compiler is used, and there is no overhead as the expression
engine is not needed at runtime. The expression above would be compiled to:

$this->get("bar")->getvalue($this->getParameter("value"))

Using Expression for Security Access Control Rules

The second use case in Symfony is for access rules.

As we all know, the way to configure the security access control rules is confusing, which might lead to insecure applications (see http://symfony.com/blog/security-access-control-documentation-issue for more information).

Here is how the new allow_if works:

access_control:
    - { path: ^/_internal/secure, allow_if: "'127.0.0.1' == request.getClientIp() or has_role('ROLE_ADMIN')" }

This one restricts the URLs starting with /_internal/secure to people browsing from the localhost. Here, request is the current Request instance. In the expression, there is access to the following variables:

  • request
  • token
  • user

And to the following functions:

  • is_anonymous
  • is_authenticated
  • is_fully_authenticated
  • is_rememberme
  • has_role

You can also use expressions in Twig, which works well with the is_granted function:

{% if is_granted(expression('has_role("FOO")')) %}
   ...
{% endif %}

Using Expressions in the Routing

Out of the box, Symfony can only match an incoming request based on some pre-determined variables (like the path info, the method, the scheme, ...). But some people want to be able to match on more complex logic, based on other information of the Request object. That's why we introduced RequestMatcherInterface recently (but we no default implementation in Symfony itself).

The first change I've made (not related to expression support) is implement this interface for the default UrlMatcher. It was simple enough.

Then, I've added a new condition configuration for Route objects, which allow you to add any valid expression. An expression has access to the request and to the routing context.

Here is how one would configure it in a YAML file:

hello:
    path: /hello/{name}
    condition: "context.getMethod() in ['GET', 'HEAD'] and request.headers.get('User-Agent') =~ '/firefox/i'"

Why do I keep the context as all the data are also available in the request? Because you can also use the condition without using the RequestMatcherInterface, in which case, you don't have access to the request. So, the previous example is equivalent to:

hello:
    path: /hello/{name}
    condition: "request.getMethod() in ['GET', 'HEAD'] and request.headers.get('User-Agent') =~ '/firefox/i'"

When using the PHP dumper, there is no overhead as the condition is compiled. Here is how it looks like:

// hello
if (0 === strpos($pathinfo, '/hello') && preg_match('#^/hello/(?P<name>[^/]++)$#s', $pathinfo, $matches) && (in_array($context->getMethod(), array(0 => "GET", 1 => "HEAD")) && preg_match("/firefox/i", $request->headers->get("User-Agent")))) {
    return $this->mergeDefaults(array_replace($matches, array('_route' => 'hello')), array ());
}

Be warned that conditions are not taken into account when generating a URL.

Using Expressions in the Validator

There is a new Expression constraint that you can put on a class. The expression is then evaluated for validation:

use Symfony\Component\Validator\Constraints as Assert;

/**
 * @Assert\Condition(condition="this.getFoo() == 'fo'", message="Not good!")
 */
class Obj
{
    public function getFoo()
    {
        return 'foo';
    }
}

In the expression, you get access to the current object via the this variable.

Dynamic annotations

The expression language component is also very useful in annotations. the SensoLabs FrameworkExtraBundle leverages this possibility to implement HTTP validation caching in the @Cache annotation and to add a new @Security annotation (see sensiolabs/SensioFrameworkExtraBundle#238.)

@lsmith77

ExpressionEngine is a slightly unfortunate name: http://ellislab.com/expressionengine
And did you see my reference to the Hateoas lib PR using https://github.com/Kitano/php-expression (which in turn is based on https://github.com/schmittjoh/serializer)?

@fabpot
Symfony member

@lsmith77 Well, if you have a look at the implementation of the code you mention, I guess they also got inspiration from Twig but my implementation is more generic and way more powerful.

@fabpot
Symfony member

For the component name, I'm open to any other suggestion.

@Taluu

For the XML syntax, wouldn't the " pose a problem ? Maybe use a dedicated <argument> tag instead to fill up the value (which would then be interpreted as a Twig expression, as this PR is meant to do so) ?

@fabpot
Symfony member

@Taluu I mentioned Twig as a reference as the code comes from there, but it's far from being a Twig template, just the expression part of it. Moving the expression as the tag value should indeed be possible:

    <service id="foo" class="Foo">
        <argument type="expression">
            service("bar").getvalue(parameter("value"))
        </argument>
    </service>
@stof stof and 2 others commented on an outdated diff Sep 2, 2013
...mponent/DependencyInjection/Loader/YamlFileLoader.php
@@ -311,6 +312,8 @@ private function resolveServices($value)
{
if (is_array($value)) {
$value = array_map(array($this, 'resolveServices'), $value);
+ } elseif (is_string($value) && 0 === strpos($value, '#')) {
@stof
stof Sep 2, 2013

# is an unfortunate choice IMO. It forces to enclose the string in quotes otherwise it is parsed as a comment (and it will be hard to debug in such case)

@fabpot
fabpot Sep 2, 2013

As I said, I've used the first character that came. A better one must be found...

@Crell
Crell Sep 2, 2013

What about adopting @ followed by some other magic char as the convention for special meaning? So @$ or @& or whatever can mean "expression", @ with no prefix means "some other service', etc. You know @X always means "Something special". (Not sure if this is possible at this point, but figured I'd throw it out there.)

@fabpot
fabpot Sep 2, 2013

I've updated the PR with @#.

@stof stof and 1 other commented on an outdated diff Sep 2, 2013
src/Symfony/Component/ExpressionEngine/Compiler.php
+ *
+ * @author Fabien Potencier <fabien@symfony.com>
+ */
+class Compiler
+{
+ private $source;
+ private $functions;
+
+ public function __construct(array $functions)
+ {
+ $this->functions = $functions;
+ }
+
+ public function getFunction($name)
+ {
+ return $this->functions[$name];
@stof
stof Sep 2, 2013

shouldn't you check for existence to avoid notices ?

@fabpot
fabpot Sep 2, 2013

it always exists at this stage as the check is done elsewhere

@stof stof commented on an outdated diff Sep 2, 2013
src/Symfony/Component/ExpressionEngine/Compiler.php
+
+ public function reset()
+ {
+ $this->source = '';
+
+ return $this;
+ }
+
+ /**
+ * Compiles a node.
+ *
+ * @param Node\Node $node The node to compile
+ *
+ * @return Compiler The current compiler instance
+ */
+ public function compile(Node\Node $node)
@stof
stof Sep 2, 2013

you should add a use statement for consistency with the codebase

@stof stof commented on an outdated diff Sep 2, 2013
src/Symfony/Component/ExpressionEngine/.gitignore
@@ -0,0 +1,5 @@
+vendor/
+composer.lock
+phpunit.xml
+Tests/ProjectContainer.php
+Tests/classes.map
@stof
stof Sep 2, 2013

these seem unused (copy paste from DI I guess)

@stof stof commented on an outdated diff Sep 2, 2013
...mfony/Component/ExpressionEngine/ExpressionEngine.php
+ return $this->parser;
+ }
+
+ public function getCompiler()
+ {
+ if (null === $this->compiler) {
+ $this->compiler = new Compiler($this->functions);
+ }
+
+ return $this->compiler->reset();
+ }
+
+ protected function registerFunctions()
+ {
+ $this->addFunction('constant', function ($constant, $object = null) {
+ return $object ? sprintf('constant(get_class(%s)."::".%s)', $object, $constant) : sprintf('constant(%s)', $constant);
@stof
stof Sep 2, 2013

Does this really expect $object to be a string ?

And the resolution of null at compile time looks wrong to me. the argument may use a variable being null or not null.

@stof
Symfony member

The replace directive is missing in the main composer.json

@stof stof commented on an outdated diff Sep 2, 2013
src/Symfony/Component/ExpressionEngine/Lexer.php
+ $this->operatorRegex = $this->getOperatorRegex();
+ $end = strlen($this->expression);
+
+ while ($this->cursor < $end) {
+ $this->lexExpression();
+ }
+
+ $this->tokens[] = new Token(Token::EOF_TYPE, null, $this->cursor);
+
+ if (!empty($this->brackets)) {
+ list($expect, $cursor) = array_pop($this->brackets);
+ throw new SyntaxError(sprintf('Unclosed "%s"', $expect), $cursor);
+ }
+
+ if (isset($mbEncoding)) {
+ mb_internal_encoding($mbEncoding);
@stof
stof Sep 2, 2013

you also need to reset it in case of an exception being thrown

@stof
stof Sep 2, 2013

This would be a perfect use case for the new PHP 5.5 finally keyword btw...

@stof stof and 1 other commented on an outdated diff Sep 2, 2013
src/Symfony/Component/ExpressionEngine/Lexer.php
+ if (preg_match(self::REGEX_STRING, $this->expression, $match, null, $this->cursor)) {
+ $this->tokens[] = new Token(Token::STRING_TYPE, stripcslashes(substr($match[0], 1, -1)), $this->cursor);
+ $this->cursor += strlen($match[0]);
+
+ return;
+ }
+
+ // unlexable
+ throw new SyntaxError(sprintf('Unexpected character "%s"', $this->expression[$this->cursor]), $this->cursor);
+ }
+
+ private function getOperatorRegex()
+ {
+ $operators = array(
+ 'not', '-', '+',
+ 'or', 'and', 'b-or', 'b-xor', 'b-and', '==', '!=', '<', '>', '>=', '<=', 'not in', 'in', '..', '+', '-', '~', '*', '/', '%', '**',
@stof
stof Sep 2, 2013

As this expression engine is meantot be used by devs, I think supporting === and !== makes sense (they are familiar with it). this is even more important as we don't have the Twig sameas test to replace it

@stof stof and 1 other commented on an outdated diff Sep 2, 2013
...ymfony/Component/ExpressionEngine/Node/BinaryNode.php
+ ->raw('(')
+ ->compile($this->nodes['left'])
+ ->raw(' ')
+ ->raw($operator)
+ ->raw(' ')
+ ->compile($this->nodes['right'])
+ ->raw(')')
+ ;
+ }
+
+ public function evaluate(array $functions, array $variables)
+ {
+ $operator = $this->attributes['operator'];
+
+ if (isset($this->functions[$operator])) {
+ return call_user_func($this->functions[$operator], $this->nodes['left']->evaluate($functions, $variables), $this->nodes['right']->evaluate($functions, $variables));
@stof
stof Sep 2, 2013

this will break for not in as !in_array is not a function. Wouldn't it be easier to compile not in(foo, bar) as not(in(foo, bar)) as we already have the not unary operator anyway ?

@stof stof and 1 other commented on an outdated diff Sep 2, 2013
...fony/Component/ExpressionEngine/Node/FunctionNode.php
+ $arguments[] = $compiler->subcompile($node);
+ }
+
+ $function = $compiler->getFunction($this->attributes['name']);
+
+ $compiler->raw(call_user_func_array($function['compiler'], $arguments));
+ }
+
+ public function evaluate(array $functions, array $variables)
+ {
+ $arguments = array($variables);
+ foreach ($this->nodes['arguments']->nodes as $node) {
+ $arguments[] = $node->evaluate($functions, $variables);
+ }
+
+ return call_user_func_array($functions[$this->attributes['name']]['evaluator'], $arguments);
@stof
stof Sep 2, 2013

you need to check the function existence here to avoid errors

@fabpot
fabpot Sep 2, 2013

Again, it's done in the Parser, so that won't happen.

@stof stof and 1 other commented on an outdated diff Sep 2, 2013
src/Symfony/Component/ExpressionEngine/Parser.php
+ switch ($token->type) {
+ case Token::NAME_TYPE:
+ $this->stream->next();
+ switch ($token->value) {
+ case 'true':
+ case 'TRUE':
+ $node = new Node\ConstantNode(true);
+ break;
+
+ case 'false':
+ case 'FALSE':
+ $node = new Node\ConstantNode(false);
+ break;
+
+ case 'none':
+ case 'NONE':
@stof
stof Sep 2, 2013

I know Twig supports none as an alias for null because of its Python inspiration, but does it make sense to keep it in this ExpressionEngine ?

@fabpot
fabpot Sep 2, 2013

I don't know yet if we need to keep it close to Twig or not at all. That's why I kept none at first, but as both will probably diverge even more than today, that's probably safe to remove none support.

@fabpot
fabpot Sep 2, 2013

ok, I've removed none and I've changed the operators to the PHP operators.

@stof stof commented on an outdated diff Sep 2, 2013
src/Symfony/Component/ExpressionEngine/Parser.php
+ } elseif ($token->test(Token::PUNCTUATION_TYPE, '{')) {
+ $node = $this->parseHashExpression();
+ } else {
+ throw new SyntaxError(sprintf('Unexpected token "%s" of value "%s"', $token->type, $token->value), $token->cursor);
+ }
+ }
+
+ return $this->parsePostfixExpression($node);
+ }
+
+ public function parseStringExpression()
+ {
+ $nodes = array();
+ // a string cannot be followed by another string in a single expression
+ $nextCanBeString = true;
+ while (true) {
@stof
stof Sep 2, 2013

Is this infinite loop really needed ? It will stop at either the first or second iteration (and doing nothing at the iteration where it breaks) because of the condition on $nextCanBeString

@stof stof commented on an outdated diff Sep 2, 2013
src/Symfony/Component/ExpressionEngine/composer.json
+ {
+ "name": "Fabien Potencier",
+ "email": "fabien@symfony.com"
+ },
+ {
+ "name": "Symfony Community",
+ "homepage": "http://symfony.com/contributors"
+ }
+ ],
+ "require": {
+ "php": ">=5.3.3"
+ },
+ "autoload": {
+ "psr-0": { "Symfony\\Component\\ExpressionLanguage\\": "" }
+ },
+ "target-dir": "Symfony/Component/ExpressionLanguage",
@stof
stof Sep 2, 2013

This composer.json is not consistent with the current namespace

@stof stof commented on an outdated diff Sep 2, 2013
src/Symfony/Component/ExpressionEngine/Parser.php
+ }
+
+ return $this->parsePrimaryExpression();
+ }
+
+ protected function parseConditionalExpression($expr)
+ {
+ while ($this->stream->current->test(Token::PUNCTUATION_TYPE, '?')) {
+ $this->stream->next();
+ if (!$this->stream->current->test(Token::PUNCTUATION_TYPE, ':')) {
+ $expr2 = $this->parseExpression();
+ if ($this->stream->current->test(Token::PUNCTUATION_TYPE, ':')) {
+ $this->stream->next();
+ $expr3 = $this->parseExpression();
+ } else {
+ $expr3 = new Node\ConstantNode('');
@stof
stof Sep 2, 2013

In case of PHP code, I think null makes more sense than an empty string (actually, I also think the same for Twig but it makes less difference as it will achieve the same output when the ternary operator is used in a print node)

@stof stof and 2 others commented on an outdated diff Sep 2, 2013
src/Symfony/Component/ExpressionEngine/Parser.php
+
+ protected function parseConditionalExpression($expr)
+ {
+ while ($this->stream->current->test(Token::PUNCTUATION_TYPE, '?')) {
+ $this->stream->next();
+ if (!$this->stream->current->test(Token::PUNCTUATION_TYPE, ':')) {
+ $expr2 = $this->parseExpression();
+ if ($this->stream->current->test(Token::PUNCTUATION_TYPE, ':')) {
+ $this->stream->next();
+ $expr3 = $this->parseExpression();
+ } else {
+ $expr3 = new Node\ConstantNode('');
+ }
+ } else {
+ $this->stream->next();
+ $expr2 = $expr;
@stof
stof Sep 2, 2013

I think the ternary operator should compile to $expr1 ?: $expr3 rather than $expr1 ? $expr1 : $expr3 in this case to let PHP optimize the evalutation (not sure it does such optimization currently, but it seems sane to assume it could at least)

@stof
stof Sep 2, 2013

note that this advice could maybe be applied to Twig as well.

@fabpot
fabpot Sep 2, 2013

Twig supports PHP 5.2, which does not allow for the a ?: b construct.

@stof
stof Sep 2, 2013

Well, maybe this optimization could be used in Twig for 5.3+, still compiling with the duplicate expression for 5.2. there is already some optimized compilation for 5.4+ IIRC (or at least there used to be)

@Crell
Crell Sep 2, 2013

Is the intent for Twig to use this library as well? I don't see how that's possible since it's namespaced. If not we may as well support the full 5.3 syntax.

@fabpot
fabpot Sep 2, 2013

If you mean, replace the Twig expression sub-system by this code, the answer is no as Twig is much more powerful.

@stof
Symfony member

@Taluu I agree that putting the expression in the value can make it easier, but it is already possible to use quotes in the expression when used in the XML attribute. All you need to do to avoid nasty escaping (mixing the XML escaping and the ExpressionEngine one will look ugly quickly) is to use different quotes as both ExpressionEngine and XML allows you to choose between single quotes and double quotes (the same is true for the YAML case btw). I fixed the example to use it.

@Taluu

Still, even if it is possible, I still think it would be way more readable in the value rather the attribute. Or why not do both (for simple use case, use the attribute, and the body it is complicated) ?

@stof
Symfony member

@Taluu I think we should keep a single way to do it for consistency and ease to document it.

I would vote for the tag value. It would indeed be easier to write quotes (but it will require escaping < in XML) and will be consistent with the way arguments are specified with type="string" (default) or type="constant"

@stof stof commented on an outdated diff Sep 2, 2013
src/Symfony/Component/ExpressionEngine/Compiler.php
+ }
+
+ $this->raw($value);
+
+ if (false !== $locale) {
+ setlocale(LC_NUMERIC, $locale);
+ }
+ } elseif (null === $value) {
+ $this->raw('null');
+ } elseif (is_bool($value)) {
+ $this->raw($value ? 'true' : 'false');
+ } elseif (is_array($value)) {
+ $this->raw('array(');
+ $i = 0;
+ foreach ($value as $key => $value) {
+ if ($i++) {
@stof
stof Sep 2, 2013

using a boolean to detect the first element would be clearer than the incrementing index used just for this IMO. thus, it would match the other parts of the component dumping a list

@stof
Symfony member

@fabpot Just a consideration when implementing the support of expressions for access control rules. You should try to avoid incompatibilities with the expression engine of JMSSecurityExtraBundle to avoid breaking the existing apps. This means choosing a different key than expression in the access_rules config in YAML.
And it would be awesome to allow passing the expression as a simple string to is_granted instead of passing an object wrapping the expression, to let the expression work directly in any place allowing to configure the role name used for the security check (which is something JMSSecurityExtraBundle failed to achieve, requiring such bundles to explicitly integrate with it).

@stof
Symfony member

For the Yaml format, I hate more and more this prefix-based syntax. Each time we want to add a new feature, we would have to break BC by making the new prefix a reserved character in first position, requiring to escape it for normal use (before 2.3, we forgot this need to escape @ at the beginning of string arguments). Unfortunately, I don't really see a good way to avoid this (we cannot have XML attributes in YAML to distinguish different argument types). Whatever char we choose to identify an expression will be a BC break.

@fabpot
Symfony member

@stof I do agree. The YAML format is far from being explicit with all kind of possible prefixes like @, @@, @!, and probably more I don't remember right now. Using 2 characters can mitigate the problem though, even if not perfect.

@fabpot
Symfony member

@stof you mean the Twig is_granted function?

@benjamindulau

@fabpot In what way your component is more powerful than the one we extracted from JMSSerializerBundle ? see: https://github.com/Kitano/php-expression

For a real case usage see our cache-bundle at https://github.com/Kitano/KitanoCacheBundle

As @lsmith77 mentioned, a PR is also open for using it in the @willdurand's HATEOAS lib here: willdurand/Hateoas#54

We think that @schmittjoh component was powerful enough and that with some improvements, the library we extracted from it could easily be production ready.

@stof
Symfony member

@fabpot Both the Twig function and the isGranted function of the security context actually

@fabpot
Symfony member

@benjamindulau The code you are talking about comes from Twig, which I wrote. And then, it was changed and strip-down by various people. The code from Johannes is very specific and licensed under the Apache license. I won't even talk about the other repo as it claims to be MIT but the most of the code is borrowed from Johannes which is not MIT.

For the supported features, I've listed them in the summary of this PR.

@stof
Symfony member

@benjamindulau It does not come from the JMSSerializerBundle but from JMSSecurityExtraBundle.

and the big advantage of this engine IMO is the possibility to run in 2 modes: evaluation or compilation. the JMS engine only supports compilation (and then relies on eval to provide evaluation when you don't cache them).
thus, the compilation produces a closure, not runnable code, so using an expression would still imply some runtime overhead in the container.

and indeed, your library is a huge mess concerning licensing. You tell in your README that it is MIT licensed but most of your file still contain the Apache2 license of the original library, which means they are not under MIT at all (IIRC, the license mentioned in the file wins over the LICENSE file shipped in the project)

@fabpot
Symfony member

As @stof said:

  • complete implementation of basics operators (with a proven precedence algorithm);
  • no need to use eval;
  • possibility to use the evaluator or the compiler;
  • return a single PHP string that can be easily embedded elsewhere.
@fabpot
Symfony member

@stof Why do you think that it is a problem to pass an object? For the PHP version, I don't see that as a problem and for the Twig tag, we can do something like is_granted(expression('...')). Sounds simple enough, no?

@stof
Symfony member

@fabpot It is an issue for bundles like SonataAdminBundle or FOSCommentBundle allowing to change the role being checked through their config. It is not possible to pass an Expression in there from the config file. So these bundles would have to provide a separate way to configure an expression in this case, either through a separate key (the cleaner way) or through a naming convention (inspired by the way the Yaml DI loader works, which is what TwigBundle does for its globals). The Config component works for scalars and arrays in the config, not really for value objects wrapping a scalar (and Yaml does not either).
For Twig, it could indeed be simple enough (and for PHP too probably if the Expression class does not implement Traversable like the JMS one, thus causing issues with the array casting). Additionnally, in Twig, the expression function could eventually be compiled in an optimized way by doing it when compiling the Twig template (it may require customizing the way variables are compiled in this case though as they will more likely be available in an array than like different PHP variables with the given name)

@benjamindulau

@stof and @fabpot: Right, the licenses are a mess, but that's not a problem without solution ;-)

As for the features, the 2 modes possibility is nice indeed.

The future of this proposal is important for us, because there is no point in maintaining our code if an official symfony-branded component is created. We'd better contributing to it instead of doing exactly the same separately.
It's just unfortunate that you didn't try to contact us, or even @schmittjoh (maybe you did) in order to mutualise our effort. But hey, such is life!

Concerning the name of the component, if this component really is to be created and officially supported, I will definitely vote for a more "official" name like "Php Expression Language (PEL)" or maybe "Symfony Expression Language (SEL or SfEl)" if it's very specific to Symfony, just like the one for Spring in the java world.

@benjamindulau

@fabpot

An open question is whether we keep the difference betweens arrays and hashes.

IMO, since this more or less a PHP expression language, the expressions should try to match exactly the PHP syntax everytime it's possible. So I'd vote for no differences.

@fabpot
Symfony member

@benjamindulau: I've been discussing this matter with Johannes for quite a few months now, but I did not really had a look at his code; but when I did this week-end, I realized that I don't need to ask anyone to hack my own code, and so I created this new component.

@schmittjoh
@benjamindulau

@fabpot still, there is the code, but also the intention behind it that count. For the code, that's what open source is. Maybe @schmittjoh took inspiration from your Twig library code (and to be honest, i'm not sure that's what he did), then we took @schmittjoh code and changed it a bit again, but that's the exercice of the open source I think.

We didn't have the feeling that we were hacking your code, we didn't even realized that.

Now for the intention, meaning why we decided to extract the code and make a standalone library, that's one thing we could have discussed together, without any regard to the quality of the code itself (or even the licenses).

I don't want to start a big debate about all this, I was just trying to make a point.

Above all this, I'm glad such a component is about to be promoted, because it's a very powerful one with many use cases.

@stof
Symfony member

@benjamindulau in your case, you should have discussed the license matter. Relicensing Apache code to MIT is not allowed by the license (the opposite is) so it requires you to ask the permission of all contributors to the JMSSecurityExtraBundle expression system first. Otherwise, you are violating the license.

@schmittjoh
@henrikbjorn

Personally i think this is a very powerful and useful component. I think it is important that we advocate some of the use cases for this. So we don't end up with all a bunch of TypoScript type of libraries.

👍

@Crell Crell and 1 other commented on an outdated diff Sep 2, 2013
src/Symfony/Component/ExpressionEngine/README.md
@@ -0,0 +1,36 @@
+ExpressionEngine Component
+==========================
+
+ExpressionEngine provides an engine that can compile and evaluate expressions.
+
+ $engine = new ExpressionEngine();
+
+ echo $engine->evaluate('1 + foo', array('foo' => 2));
+
+ echo $engine->compile('1 + foo');
@Crell
Crell Sep 2, 2013

As the PHP manual does, it would be helpful to show what the expected output of these calls is. That would help drive home what the value of this library is. (Especially the compiling part.) The same for the extended version below, which is arguably even more valuable.

@Crell

Oh this hurts my brain... But I can definitely see the benefit for the DIC, which would help Drupal.

As for other uses, the OP mentions access rules as a possibility. I've not worked with the Security component, but if I read this correctly it means one could put complex boolean logic into, say, a route definition to allow access to a route "if these two things are true or if this one other thing is true". Correct? (This is another use case that Drupal has right now where we have a simple "any" or "all" toggle, but we've had requests for more complex logic.)

@adrienbrault adrienbrault commented on an outdated diff Sep 2, 2013
...ny/Component/DependencyInjection/Dumper/PhpDumper.php
@@ -1197,6 +1200,8 @@ private function dumpValue($value, $interpolate = true)
}
return $this->getServiceCall((string) $value, $value);
+ } elseif ($value instanceof Expression) {
+ return $this->getExpressionEngine()->compile($value->getExpression());
@adrienbrault
adrienbrault Sep 2, 2013

Why using a getter here and not in the ContainerBuilder ?

@adrienbrault adrienbrault commented on an outdated diff Sep 2, 2013
...mponent/DependencyInjection/Loader/YamlFileLoader.php
@@ -311,6 +312,8 @@ private function resolveServices($value)
{
if (is_array($value)) {
$value = array_map(array($this, 'resolveServices'), $value);
+ } elseif (is_string($value) && 0 === strpos($value, '@#')) {
@adrienbrault
adrienbrault Sep 2, 2013

What about @= ?

@bobdenotter

Well, if you have a look at the implementation of the code you mention, I guess they also got inspiration from Twig but my implementation is more generic and way more powerful.

I agree that Expression Engine is a very unfortunate name. The CMS “Expression Engine” has been around since 2002, I think. So, that's way longer than Twig.

@adrienbrault

Really cool stuff 👍

@boonkerz

math should use bcmath when it's available.

in reference to http://php.net/manual/de/language.types.float.php

@beryllium

Per @fabpot's request on twitter, I suggested that Symfony Evaluator Component might be a candidate for a new name. Another one could be "DiscretionEngine". Since it's based on Twig, I thought maybe another tree-related noun would work, but as this is being proposed as an actual component it seems like the tree-noun idea wouldn't fit the component naming pattern.

@benjamindulau

What about simply Symfony\Expression ?

@stof stof and 1 other commented on an outdated diff Sep 3, 2013
...ymfony/Component/ExpressionEngine/Node/BinaryNode.php
+ public function evaluate(array $functions, array $variables)
+ {
+ $operator = $this->attributes['operator'];
+
+ if (isset($this->functions[$operator])) {
+ $left = $this->nodes['left']->evaluate($functions, $variables);
+ $right = $this->nodes['right']->evaluate($functions, $variables);
+ if ('not in' == $operator) {
+ return !call_user_func('in_array', $left, $right);
+ }
+
+ return call_user_func($this->functions[$operator], $left, $right);
+ }
+
+ $left = $this->nodes['left']->evaluate($functions, $variables);
+ $right = $this->nodes['right']->evaluate($functions, $variables);
@stof
stof Sep 3, 2013

you could move this before the isset($this->functions[$operator]) condition as you have exactly the same code inside it

@stof stof and 1 other commented on an outdated diff Sep 3, 2013
src/Symfony/Component/ExpressionEngine/Node/NameNode.php
+
+class NameNode extends Node
+{
+ public function __construct($name)
+ {
+ $this->attributes = array('name' => $name);
+ }
+
+ public function compile(Compiler $compiler)
+ {
+ $compiler->raw('$'.$this->attributes['name']);
+ }
+
+ public function evaluate(array $functions, array $variables)
+ {
+ return $variables[$this->attributes['name']];
@stof
stof Sep 3, 2013

shouldn't you check whether the variable is defined ?

@fabpot
fabpot Sep 3, 2013

the variable names are now checked in the Parser.

@stof stof and 1 other commented on an outdated diff Sep 3, 2013
src/Symfony/Component/ExpressionEngine/Node/NameNode.php
+
+namespace Symfony\Component\ExpressionEngine\Node;
+
+use Symfony\Component\ExpressionEngine\Compiler;
+use Symfony\Component\ExpressionEngine\SyntaxError;
+
+class NameNode extends Node
+{
+ public function __construct($name)
+ {
+ $this->attributes = array('name' => $name);
+ }
+
+ public function compile(Compiler $compiler)
+ {
+ $compiler->raw('$'.$this->attributes['name']);
@stof
stof Sep 3, 2013

This looks weird to me. It does not allow validating variable names at all and depends of the place where the compiled expression is evaluated

@fabpot
fabpot Sep 3, 2013

the variable names are now checked in the Parser.

@fabpot
fabpot Sep 3, 2013

and the valid names must be passed to the compile() method. The goal being to not do anything special in the compiled string.

@makasim

I have doubts it is worth it. Current DI implementation is strict and clean, and I like it. I am sure I do not get any surprises with it. Now, with expressions a developer may write whatever he wants, possibly a bad\buggy code.

It was very few times when I really wanted something like this and to be honest all those time there are was a solution.

@benjamindulau

@makasim There are many more use cases than just in the DIC configuration. As always, that's not the component the issue, but what we do with it. I guess Symfony core devs will be reasonable about that and won't use it anywhere and anyhow.

@fabpot
Symfony member

I've renamed the component to ExpressionLanguage. I did not choose to use the shorter Expression name as qualifying expression is probably important, and because I needed a more descriptive name for the main class as well. The new name works quite well when used in code:

use Symfony\Component\ExpressionLanguage\ExpressionLanguage;

$language = new ExpressionLanguage();

$language->addFunction();
$language->evaluate();
$language->compile();
@stof stof commented on an outdated diff Sep 3, 2013
src/Symfony/Component/ExpressionLanguage/composer.json
+ {
+ "name": "Fabien Potencier",
+ "email": "fabien@symfony.com"
+ },
+ {
+ "name": "Symfony Community",
+ "homepage": "http://symfony.com/contributors"
+ }
+ ],
+ "require": {
+ "php": ">=5.3.3"
+ },
+ "autoload": {
+ "psr-0": { "Symfony\\Component\\ExpressionLanguage\\": "" }
+ },
+ "target-dir": "Symfony/Component/Expression",
@stof
stof Sep 3, 2013

the target dir is wrong

@stof stof commented on an outdated diff Sep 3, 2013
...ny/Component/DependencyInjection/Dumper/PhpDumper.php
@@ -1197,6 +1200,8 @@ private function dumpValue($value, $interpolate = true)
}
return $this->getServiceCall((string) $value, $value);
+ } elseif ($value instanceof Expression) {
+ return $this->getExpressionLanguage()->compile($value->getExpression());
@stof
stof Sep 3, 2013

you are missing the allowed names here

@Taluu Taluu commented on an outdated diff Sep 3, 2013
src/Symfony/Component/ExpressionLanguage/Lexer.php
+
+ private function getOperatorRegex()
+ {
+ $operators = array(
+ 'not', '!', '-', '+',
+ 'or', '||', '&&', 'and', '|', '^', '&', '==', '===', '!=', '!==', '<', '>', '>=', '<=', 'not in', 'in', '..', '+', '-', '~', '*', '/', '%', '=~', '!~', '**',
+ );
+
+ $operators = array_combine($operators, array_map('strlen', $operators));
+ arsort($operators);
+
+ $regex = array();
+ foreach ($operators as $operator => $length) {
+ // an operator that ends with a character must be followed by
+ // a whitespace or a parenthesis
+ $regex[] = preg_quote($operator, '/').(ctype_alpha($operator[$length - 1]) ? '(?=[\s()])' : '');
@Taluu
Taluu Sep 3, 2013

With this regex, IIRC, you may match things like and ) (operator ending with a character followed by a closing parenthesis), no ? I don't think this is really what you intended to match ?

@webmozart
Symfony member

I haven't checked the code yet, but I like what I see! I can see one more use case for this, which is validation:

/**
 * @Assert\Expression("value() <= this.getMaxNumberOfLoginAttempts()", message="Too many login attempts.")
 */
private $loginAttempts;

A very strong benefit of this approach is that we can compile the same expression into JavaScript.

Having said that, I think it is important to support the concept of variables, such as ExpressionEngine does. Variables would facilitate both the use in the DI container and in the validator. I suggest to use the same syntax we use now, %var_name%.

<parameters>
    <parameter key="value">foobar</parameter>
</parameters>
<services>
    <service id="foo" class="Foo">
        <argument type="expression" value="service('bar').getValue(%value%)" />
    </service>
    <service id="bar" class="Bar" />
</services>

I think that people will actually expect to be able to access parameters like that from within expressions. Meeting their expectation would be a big UX plus.

/**
 * @Assert\Expression("%value% <= this.getMaxNumberOfLoginAttempts()", message="Too many login attempts.")
 */
private $loginAttempts;

When compiling such code, variables could either be turned into PHP variables (which obviously need to be imported into the current scope before executing the compiled code) or into accesses to a VariableBag instance which must be made available before execution.

$value <= $this->getMaxNumberOfLoginAttempts()

or

$variableBag->get('value') <= $this->getMaxNumberOfLoginAttempts()
@webmozart
Symfony member

btw I think the name ExpressionLanguage is very good 👍

@fabpot
Symfony member

@bschussek The ExpressionLanguage component should be agnostic and as you said the fact that it is very close to JavaScript is a big plus. We definitely have support for variables. In the DIC, if you want to access a parameter, you can do so via parameter('foo').

For the validation, you would do something like:

/**
 * @Assert\Expression("value <= this.getMaxNumberOfLoginAttempts()", message="Too many login attempts.")
 */
private $loginAttempts;
@webmozart
Symfony member

@fabpot I agree that the component should be agnostic, but still I have to play the consistency terrorist card here. Imagine the following situation:

<parameters>
    <parameter key="my_mailer.class">Acme\HelloBundle\Mailer</parameter>
    <parameter key="my_mailer.transport">sendmail</parameter>
</parameters>

<services>
    <service id="my_mailer" class="%my_mailer.class%">
        <argument>%my_mailer.transport%</argument>
        <argument type="expression" value="parameter('my_mailer.transport') == 'foo' ? 'bar' : 'baz'" />
    </service>
</services>

This is unacceptable. We keep on adding a new way of doing the same thing with every feature we add, only for technical reasons (agnosticism in this case), completely ignoring DX. Please let's not do that! I'm sure we find a solution if we try a little harder.

@stof
Symfony member

@bschussek Using expression language variable to access DI parameters looks wrong to me for several reasons:

  • the expression language is validating the variable names, so it would require knowing all parameter names upfront
  • passing all parameters as variable to the evaluation will have a cost (and it will require resolving all parameters for it even if they are not actually needed)
  • it will be really easy to conflict with other variables that you might want to pass to the expression (for instance this for the container itself).

Using a function to get the parameter seems more logical (and when you are writing your code, you cannot use %parameter% either to get the parameter)

Note that if we follow your reasoning further, we would also conclude that accessing services in an expression should use the @id notation when being in a YAML file because it is how YAML files are referencing it (and I don't know what else would be used when getting the expression from other sources)

@webmozart
Symfony member

@stof With a little creativity you can solve all of these problems ;) Again, we could introduce the concept of global parameters to the ExpressionLanguage, either as extension (only available for the DIC) or native. The parameters could be provided by accessing an implementation of a simple ParameterBagInterface.

interface ParameterBagInterface
{
    public function get($parameter);
}

Depending on the use case, parameters can be chosen to be supported or not. For example, the DIC could support parameters, the validator not (to simplify translating the expressions to JavaScript) - simply by passing a bag or leaving it null.

The issue with services accessed via @service_id is fortunately not as grave, because it is restricted to YAML. Not sure whether it is absolutely necessary to find a solution for that (although it would certainly be nice to provide the same functionality through an extension to the language).

@stof
Symfony member

@bschussek As I don't like the prefixed-based references in YAML as said above (but I don't see a better way to do it), I won't spend time trying to use a similar syntax in expressions where we can do it in a more readable way through a function IMO.

Regarding the use of % to reference parameters in expressions, I think it might cause confusions as the modulo operator is also a %

@fabpot
Symfony member

I've started the implementation of expressions in the new access control rules. Read the "Using Expression for Security Access Control Rules" section in the PR description for more information. The allow_if keyword is up to the discussion (access being taken by the JMS bundle, we need to find something else).

@mvrhov

Well if we can come up with a way to replace the jms bundle in core then I can't see nothing wrong with reusing their keyword

@fabpot
Symfony member

@Crell I've just created a Gist that demonstrates how you could very easily use the expression language to add conditions in the Symfony routing: https://gist.github.com/fabpot/6433461

@sstok

The routing example seems very cool.

Would be nice to have this for default-value as well, as it solves some problems with using host-match and not always passing in the value. #6857

@lsmith77

could routing support be a solution to get media type based versioning covered?

@fabpot
Symfony member

@lsmith77 Anything based on information available on the Request would be possible.
@sstok Indeed, it would provide a very flexible routing solution

I've just committed a first version of the implementation in the Routing that works well (with the condition dumped in the PHP file, so without any overhead).

I've updated the PR description with more information about the implementation.

@stof
Symfony member

@mvrhov The issue is that this expression language is not compatible with the JMS one. So reusing the same key means that an application which is using the JMS bundle will break when upgrading to Symfony 2.4 when parsing the expression with the core component (or it will overwrite the system with the JMS one and thus break any shared bundle relying on the core behavior)

@fabpot fabpot referenced this pull request in willdurand/BazingaRestExtraBundle Sep 4, 2013
Closed

[Question] No stable version? #2

@fabpot
Symfony member

@crell With 3584c51, Symfony Router will support conditions out of the box ;)

@mvrhov

@stof: I'm aware of this. What I'm tryin to say is that I'd happily throw out an 3rd party bundle and change the annotations, if the core component would have at least the same feature set.

@stof
Symfony member

@mvrhov I never said that the core should not provide the feature. But it should use a different property name in the config so that both systems can coexist. this way, it does not break apps using the JMS expression system

@wouterj
Symfony member

This is getting more awesome every hour! :)

But why don't we use the same name for all different use cases? Having expression everywhere seems the most logical to me.

@fabpot
Symfony member

@WouterJ Thanks for having a look at consistency. I need more feedback about the names. That said, I think it does not make sense to have the same name everywhere. The name should reflect what you want to do with the expression, not that the content is an expression. For the routing, the expression is a condition for the route to match (that's why the name is condition). For the access rules, the expression defines the condition for the grant access for the path (hence the allow_if name). And for the service container, I used expression for XML as this is indeed a generic way to inject the result of the evaluation of an expression.

Anyone with better names?

@wouterj
Symfony member

imo, everything you can configure for a route (except from defaults) are a condition. But to make life easy, we have called it a path condition, a method condition, etc. Same for the access rules, everything is about 'allowing a user if ...'. That's why I suggested expression.

Something else I discovered after playing with it: Inconsistent variable names. In access rules object (a very generic name) reference to the request, but in the routing the request is available with request. What is the exact reason to use object in the access rules, can it also be something else than a request?
And why do we use this in the DI, it seems better to me to use container.

@fabpot
Symfony member

@WouterJ Great! I can see that you actually played with it! I'm not happy with object but there is no easy way around it as it comes from the expression voter, which can vote on many objects, including the request. But as I do agree with you that this is suboptimal, I'm going to push a patch/hack/whatever-you-call-it to make it possible to use request in this specific case. You're probably right for the this variable name and I'm going to rename it to container. Thanks for the feedback, much appreciated.

@1ed 1ed and 1 other commented on an outdated diff Sep 4, 2013
...nt/Security/Core/Authorization/ExpressionLanguage.php
+ return $variables['trust_resolver']->isAnonymous($variables['token']);
+ });
+
+ $this->addFunction('is_authenticated', function () {
+ return '!$trust_resolver->isAnonymous($token)';
+ }, function (array $variables) {
+ return !$variables['trust_resolver']->isAnonymous($variables['token']);
+ });
+
+ $this->addFunction('is_fully_authenticated', function () {
+ return '!$trust_resolver->isFullFledge($token)';
+ }, function (array $variables) {
+ return !$variables['trust_resolver']->isFullFledge($variables['token']);
+ });
+
+ $this->addFunction('is_rememberme', function () {
@1ed
1ed Sep 4, 2013

This should be is_remember_me, shouldn't it? Or it collides with JMSSecurity? Maybe is_remembered?

@fabpot
fabpot Sep 4, 2013

was just a typo, fixed now

@fabpot
Symfony member

@bschussek I've implemented a new constraint that takes an expression. For now, it can only be run on a class and not a property as I don't know how such a constraint on a property could have access to the whole object. Does it sounds good for you?

I've updated the PR description with some more information about how it works.

@wouterj
Symfony member

This will also be usefull when using profiler matchers

@fabpot
Symfony member

As another example of the usefulness of the expression language component, I've started to work on a new version of the SensioLabs FrameworkExtraBundle: see sensiolabs/SensioFrameworkExtraBundle#238.

@shouze

@fabpot : why not using this kind of expression engine ? https://github.com/hoaproject/Ruler

The lexer of your Expression-Language component is pretty simple even... simplistic. I don't know if it's your initial intention here.

@stephpy

@shouze +1, an expression engine based on https://github.com/hoaproject/Compiler with pp language.

@fabpot
Symfony member

@Taluu @stof I've moved the expression in the service container configuration to the node value instead of the value attribute. It is indeed much better for consistency.

@jbinfo

@fabpot you have renamed the instance of ExpressionLanguage to $language in your Gist, I think it's better to use $exprLanguage ?

@henrikbjorn

Maybe $expression would be even better.

@Crell

@fabpot: Nifty. We'd be interested in using it more for access control than routing, but I think that demonstrates that it should be pretty straightforward to implement.

It also occurs to me that we should be using the Routing YamlFileLoader directly, which I don't think we're doing. Gotta go fix that... wanders off to patch Drupal

Edit: Bah, forgot, we're not using the Config component which is why we have to write our own. :-/

@Crell

For routing purposes, would it make sense to investigate folding the existing one-off rules (method, domain, scheme, etc.) into compiling down to an expression? That is, putting all of the run-time checking into just the expression. I can see that might be faster, but we also wouldn't get as detailed a set of error cases (eg, 404 vs. 405 vs 406 errors). Not sure if that's a good idea or not, but throwing it out there.

@fabpot
Symfony member

I think I'm done with the technical stuff.

Now, here are some open questions (mostly about names!):

  • allow_if name in access control rules, do we want another one?
  • @= notation for YAML configs in the SC
  • condition name for routes
  • @Expression tag name for validation
@henrikbjorn

I would be cool if we had benchmarks for different types of expression. Parsed vs evaluated etc.

@fabpot
Symfony member
@webmozart webmozart and 1 other commented on an outdated diff Sep 7, 2013
...Symfony/Component/Validator/Constraints/Condition.php
+ * (c) Fabien Potencier <fabien@symfony.com>
+ *
+ * For the full copyright and license information, please view the LICENSE
+ * file that was distributed with this source code.
+ */
+
+namespace Symfony\Component\Validator\Constraints;
+
+use Symfony\Component\Validator\Constraint;
+
+/**
+ * @Annotation
+ *
+ * @author Fabien Potencier <fabien@symfony.com>
+ */
+class Condition extends Constraint
@webmozart
webmozart Sep 7, 2013

I don't think "Condition" is a good name here. All constraints define conditions that are to be enforced, but the manner of specifying this condition differs from constraint to constraint. So this is very ambiguous. Since the manner of writing a condition is an expression here, I'd name this constraint "Expression".

@webmozart webmozart commented on an outdated diff Sep 7, 2013
...Symfony/Component/Validator/Constraints/Condition.php
+ * file that was distributed with this source code.
+ */
+
+namespace Symfony\Component\Validator\Constraints;
+
+use Symfony\Component\Validator\Constraint;
+
+/**
+ * @Annotation
+ *
+ * @author Fabien Potencier <fabien@symfony.com>
+ */
+class Condition extends Constraint
+{
+ public $message = 'This value is not valid.';
+ public $condition;
@webmozart
webmozart Sep 7, 2013

Again, "expression".

@ruian ruian commented on an outdated diff Sep 9, 2013
src/Symfony/Component/ExpressionLanguage/Parser.php
+ switch ($token->value) {
+ case 'true':
+ case 'TRUE':
+ return new Node\ConstantNode(true);
+
+ case 'false':
+ case 'FALSE':
+ return new Node\ConstantNode(false);
+
+ case 'null':
+ case 'NULL':
+ return new Node\ConstantNode(null);
+
+ default:
+ if ('(' === $this->stream->current->value) {
+ if (false === $function = isset($this->functions[$token->value])) {
@ruian
ruian Sep 9, 2013

Why create a new variable here ?

@stof stof commented on an outdated diff Sep 9, 2013
.../DependencyInjection/Tests/Fixtures/xml/services9.xml
@@ -49,6 +49,9 @@
<call method="setBar">
<argument type="service" id="foobaz" on-invalid="ignore"/>
</call>
+ <call method="setBar">
+ <argument type="expression">service("foo") ~ parameter("foo")</argument>
@stof
stof Sep 9, 2013

I think the test should use a valid expression. Concatenating an object (the service) with a string (the parameter) is unlikely to be a valid one...

@stof stof commented on an outdated diff Sep 9, 2013
src/Symfony/Component/ExpressionLanguage/TokenStream.php
+ * file that was distributed with this source code.
+ */
+
+namespace Symfony\Component\ExpressionLanguage;
+
+/**
+ * Represents a token stream.
+ *
+ * @author Fabien Potencier <fabien@symfony.com>
+ */
+class TokenStream
+{
+ private $tokens;
+ private $position;
+
+ public $current;
@stof
stof Sep 9, 2013

you should add the phpdoc on the public property with its type.

and according to the CS, it should be defined before private properties, not after

@stof
Symfony member

A few components are missing the dev dependency and the suggestion on ExpressionLanguage (validator at least)

and I would like a way to make the expression language extendable by registering our own functions. This would be useful for reusing some login in expressions used in several places, without duplication (this would be useful for instance in the new @Security annotation of SensioFrameworkExtraBundle to avoid duplicating logic in several conditions but reusing some functions instead). Without this feature, I would not be able to replace the security expressions of JMSSecurityExtraBundle easily.

@michelsalib

@stof, do you mean something like a tag/compiler, and make the ExpressionLanguage a service inside the FrameworkBundle ? If so, I totally agree with you about this need, but IMO it could be done easily in a later PR. Let's do things one by one, this PR is already too big.

@fabpot, once again an excellent addition to the Symfony Components, I already have some use case for my personal libs, can't wait to see the PR merged! 👍

@fabpot
Symfony member

@stof Can you be more precise? You can definitely add some functions with the expression language.

@arodiss

@stof

For the Yaml format, I hate more and more this prefix-based syntax.
Each time we want to add a new feature, we would have to break BC by making the new prefix
a reserved character

Actually the solution might be to break BC NOW and only allow a-z0-9 to be first character in service name. By this you will get a good pool of non-used characters to identify upcoming features

@stof
Symfony member

@arodiss actually, we already have some restrictions on the service ids IIRC. So using @ with a second special char after it seems fine

@stof
Symfony member

@fabpot There is no way to hook into the expression parsing of the security expressions as they are parsed inside the DI extension: https://github.com/symfony/symfony/pull/8913/files#L7R618

@adrienbrault

Could the component be merged without the bundles/etc integrations ?

@michelsalib

I agree with @adrienbrault, I don't think we need all the others stuffs inside this PR and I also cannot wait to use it :)

@cordoval cordoval commented on the diff Sep 15, 2013
...fony/Bundle/FrameworkBundle/Resources/config/form.xml
@@ -54,9 +54,6 @@
<argument type="service" id="validator.mapping.class_metadata_factory" />
</service>
- <!-- PropertyAccessor -->
- <service id="property_accessor" class="%property_accessor.class%" />
@cordoval
cordoval Sep 15, 2013

if you have removed the service then please remove the class key also from this file as it is already moved to its own property_access.xml 👶

@cordoval cordoval commented on an outdated diff Sep 15, 2013
...mfony/Component/ExpressionLanguage/Node/ArrayNode.php
+ $this->index = -1;
+ }
+
+ public function addElement(Node $value, Node $key = null)
+ {
+ if (null === $key) {
+ $key = new ConstantNode(++$this->index);
+ }
+
+ array_push($this->nodes, $key, $value);
+ }
+
+ /**
+ * Compiles the node to PHP.
+ *
+ * @param Twig_Compiler A Twig_Compiler instance
@cordoval
cordoval Sep 15, 2013

should this be kept to Compiler? or is fine?

@cordoval cordoval commented on the diff Sep 15, 2013
...ony/Component/ExpressionLanguage/Node/GetAttrNode.php
@@ -0,0 +1,91 @@
+<?php
+
@cordoval
cordoval Sep 15, 2013

too many extra line breaks

@fabpot fabpot added a commit that referenced this pull request Sep 19, 2013
@fabpot fabpot merged branch fabpot/expression-engine (PR #8913)
This PR was merged into the master branch.

Discussion
----------

New Component: Expression Language

| Q             | A
| ------------- | ---
| Bug fix?      | no
| New feature?  | yes
| BC breaks?    | no
| Deprecations? | no
| Tests pass?   | yes
| Fixed tickets | #8850, #7352
| License       | MIT
| Doc PR        | not yet

TODO:

 - [ ] write documentation
 - [x] add tests for the new component
 - [x] implement expression support for access rules in the security component
 - [x] find a better character/convention for expressions in the YAML format
 - [x] check the performance of the evaluation mode
 - [x] better error messages in the evaluation mode
 - [x] add support in the Routing
 - [x] add support in the Validator

The ExpressionLanguage component provides an engine that can compile and
evaluate expressions.

An expression is a one-liner that returns a value (mostly, but not limited to, Booleans).

It is a strip-down version of Twig (only the expression part of it is
implemented.) Like Twig, the expression is lexed, parsed, and
compiled/evaluated. So, it is immune to external injections by design.

If we compare it to Twig, here are the main big differences:

 * only support for Twig expressions
 * no ambiguity for calls (foo.bar is only valid for properties, foo['bar'] is only valid for array calls, and foo.bar() is required for method calls)
 * no support for naming conventions in method calls (if the method is named getFoo(), you must use getFoo() and not foo())
 * no notion of a line for errors, but a cursor (we are mostly talking about one-liners here)
 * removed everything specific to the templating engine (like output escaping or filters)
 * no support for named arguments in method calls
 * only one extension point with functions (no possibility to define new operators, ...)
 * and probably even more I don't remember right now
 * there is no need for a runtime environment, the compiled PHP string is self-sufficient

An open question is whether we keep the difference betweens arrays and hashes.

The other big difference with Twig is that it can work in two modes (possible
because of the restrictions described above):

 * compilation: the expression is compiled to PHP and is self-sufficient
 * evaluation: the expression is evaluated without being compiled to PHP (the node tree produced by the parser can be serialized and evaluated afterwards -- so it can be saved on disk or in a database to speed up things when needed)

Let's see a simple example:

```php
$language = new ExpressionLanguage();

echo $language->evaluate('1 + 1');
// will echo 2

echo $language->compile('1 + 2');
// will echo "(1 + 2)"
```

The language supports:

 * all basic math operators (with precedence rules):
    * unary: not, !, -, +
    * binary: or, ||, and, &&, b-or, b-xor, b-and, ==, ===, !=, !==, <, >, >=, <=, not in, in, .., +, -, ~, *, /, %, **

 * all literals supported by Twig: strings, numbers, arrays (`[1, 2]`), hashes
   (`{a: "b"}`), Booleans, and null.

 * simple variables (`foo`), array accesses (`foo[1]`), property accesses
   (`foo.bar`), and method calls (`foo.bar(1, 2)`).

 * the ternary operator: `true ? true : false` (and all the shortcuts
   implemented in Twig).

 * function calls (`constant('FOO')` -- `constant` is the only built-in
   functions).

 * and of course, any combination of the above.

The compilation is better for performances as the end result is just a plain PHP string without any runtime. For the evaluation, we need to tokenize, parse, and evaluate the nodes on the fly. This can be optimized by using a `ParsedExpression` or a `SerializedParsedExpression` instead:

```php
$nodes = $language->parse($expr, $names);
$expression = new SerializedParsedExpression($expr, serialize($nodes));

// You can now store the expression in a DB for later reuse

// a SerializedParsedExpression can be evaluated like any other expressions,
// but under the hood, the lexer and the parser won't be used at all, so it''s much faster.
$language->evaluate($expression);
```
That's all folks!

I can see many use cases for this new component, and we have two use cases in
Symfony that we can implement right away.

## Using Expressions in the Service Container

The first one is expression support in the service container (it would replace
#8850) -- anywhere you can pass an argument in the service container, you can
use an expression:

```php
$c->register('foo', 'Foo')->addArgument(new Expression('bar.getvalue()'));
```

You have access to the service container via `this`:

    container.get("bar").getvalue(container.getParameter("value"))

The implementation comes with two functions that simplifies expressions
(`service()` to get a service, and `parameter` to get a parameter value). The
previous example can be simplified to:

    service("bar").getvalue(parameter("value"))

Here is how to use it in XML:

```xml
<parameters>
    <parameter key="value">foobar</parameter>
</parameters>
<services>
    <service id="foo" class="Foo">
        <argument type="expression">service('bar').getvalue(parameter('value'))</argument>
    </service>
    <service id="bar" class="Bar" />
</services>
```

and in YAML (I chose the syntax randomly ;)):

```yaml
parameters:
    value: foobar

services:
    bar:
        class: Bar

    foo:
        class: Foo
        arguments: [@=service("bar").getvalue(parameter("value"))]
```

When using the container builder, Symfony uses the evaluator, but with the PHP
dumper, the compiler is used, and there is no overhead as the expression
engine is not needed at runtime. The expression above would be compiled to:

```php
$this->get("bar")->getvalue($this->getParameter("value"))
```

## Using Expression for Security Access Control Rules

The second use case in Symfony is for access rules.

As we all know, the way to configure the security access control rules is confusing, which might lead to insecure applications (see http://symfony.com/blog/security-access-control-documentation-issue for more information).

Here is how the new `allow_if` works:

```yaml
access_control:
    - { path: ^/_internal/secure, allow_if: "'127.0.0.1' == request.getClientIp() or has_role('ROLE_ADMIN')" }
```

This one restricts the URLs starting with `/_internal/secure` to people browsing from the localhost. Here, `request` is the current Request instance. In the expression, there is access to the following variables:

 * `request`
 * `token`
 * `user`

And to the following functions:

 * `is_anonymous`
 * `is_authenticated`
 * `is_fully_authenticated`
 * `is_rememberme`
 * `has_role`

You can also use expressions in Twig, which works well with the `is_granted` function:

```jinja
{% if is_granted(expression('has_role("FOO")')) %}
   ...
{% endif %}
```

## Using Expressions in the Routing

Out of the box, Symfony can only match an incoming request based on some pre-determined variables (like the path info, the method, the scheme, ...). But some people want to be able to match on more complex logic, based on other information of the Request object. That's why we introduced `RequestMatcherInterface` recently (but we no default implementation in Symfony itself).

The first change I've made (not related to expression support) is implement this interface for the default `UrlMatcher`. It was simple enough.

Then, I've added a new `condition` configuration for Route objects, which allow you to add any valid expression. An expression has access to the `request` and to the routing `context`.

Here is how one would configure it in a YAML file:

```yaml
hello:
    path: /hello/{name}
    condition: "context.getMethod() in ['GET', 'HEAD'] and request.headers.get('User-Agent') =~ '/firefox/i'"
```

Why do I keep the context as all the data are also available in the request? Because you can also use the condition without using the RequestMatcherInterface, in which case, you don't have access to the request. So, the previous example is equivalent to:

```yaml
hello:
    path: /hello/{name}
    condition: "request.getMethod() in ['GET', 'HEAD'] and request.headers.get('User-Agent') =~ '/firefox/i'"
```

When using the PHP dumper, there is no overhead as the condition is compiled. Here is how it looks like:

```php
// hello
if (0 === strpos($pathinfo, '/hello') && preg_match('#^/hello/(?P<name>[^/]++)$#s', $pathinfo, $matches) && (in_array($context->getMethod(), array(0 => "GET", 1 => "HEAD")) && preg_match("/firefox/i", $request->headers->get("User-Agent")))) {
    return $this->mergeDefaults(array_replace($matches, array('_route' => 'hello')), array ());
}
```

Be warned that conditions are not taken into account when generating a URL.

## Using Expressions in the Validator

There is a new Expression constraint that you can put on a class. The expression is then evaluated for validation:

```php
use Symfony\Component\Validator\Constraints as Assert;

/**
 * @Assert\Condition(condition="this.getFoo() == 'fo'", message="Not good!")
 */
class Obj
{
    public function getFoo()
    {
        return 'foo';
    }
}
```

In the expression, you get access to the current object via the `this` variable.

## Dynamic annotations

The expression language component is also very useful in annotations. the SensoLabs FrameworkExtraBundle leverages this possibility to implement HTTP validation caching in the `@Cache` annotation and to add a new `@Security` annotation (see sensiolabs/SensioFrameworkExtraBundle#238.)

Commits
-------

d4ebbfd [Validator] Renamed Condition to Expression and added possibility to set it onto properties
a3b3a78 [Validator] added a constraint that runs an expression
1bcfb40 added optimized versions of expressions
984bd38 mades things more consistent for the end user
d477f15 [Routing] added support for expression conditions in routes
86ac8d7 [ExpressionLanguage] improved performance
e369d14 added a Twig extension to create Expression instances
38b7fde added support for expression in control access rules
2777ac7 [HttpFoundation] added ExpressionRequestMatcher
c25abd9 [DependencyInjection] added support for expressions in the service container
3a41781 [ExpressionLanguage] added support for regexes
9d98fa2 [ExpressionLanguage] added the component
ca62f65
@fabpot fabpot merged commit d4ebbfd into symfony:master Sep 19, 2013
@hoaproject

To continue the discussion started by @shouze and @stephpy, using Hoa\Compiler and/or Hoa\Ruler is not a bad idea. Hoa\Compiler is a compiler compiler with a dedicated grammar description language (called PP). A grammar representing your kinds of expressions is quite simple. Please, see Hoa/Math/Arithmetic.pp or Hoa/Ruler/Grammar.pp. The grammar is transformed into a compiler that can be serialized. Then, after analyzing a data (lexing + parsing), you have to exploit the resulting AST to compute a model (optional, except if you want further semantics verifications), and then, you can interprete it (the AST or the model), or you can compile it to PHP. This is the classical compiler process (please, see the french documentation of Hoa\Compiler).
As an example, please, see the README.md of Hoa\Ruler: a rule is analyzed and an AST is produced. This AST is transformed into a model thanks to an interpreter (a basic visitor). We are able to apply more visitors on this model, such as the “asserter” to really execute the rule, the “compiler” to produce PHP code, the “disassembly” to go back to the original rule. The model can also be serialized.

Please, be aware that the majority of dependencies of Hoa\Ruler can be bypassed.

We understand that Symfony may not integrate some libraries from Hoa but this is an interesting discussion and reflexion. Hoa\Compiler comes from a research study (INRIA, please, see this article) and offers a lot of services: highly hackable, lots of user-friendly errors etc. Hoa\Ruler is close to what you want to achieve.

Thoughts?

@fabpot fabpot deleted the fabpot:expression-engine branch Apr 27, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment