Userfriendly API for Namespace, Function, Class, Methods and Properties

[mneuhaus]

Hey!

i've been thinking about using your Parser to create a Tool to Manipulate Classes. This has been a common practice among the TYPO3 Community with the old kickstarter and now the ExtensionBuilder. The biggest caveat to date is the manipulation of those generated classes once the developer started to modify them. it's has been quite a buggy hassle to modfiy manually modified classes. This could simply be improved by your PHP-Parser.

To make this easier i'd propose to add some simple to use API to PHP-Parser make it easier to accomplish this task.
I've forked your Repo and started to implement some of these.

Main thing i would like to know, is if you would be willing to integrate it directly into your Parser or if you think, that this doesn't really belong directly in the Parser and instead in a seperate library.

For Example:

General

• getByType(Class|Method|Property|...)

Namespace

• getClasses()
• getFunctions()
• getName(), getType(), etc

Class

• getMethods()
• getProperties()
• addMethod()
• createMethod()
• addProperty()
• createProperty()
• getName(), getType(), etc

Method

• getArguments()
• getName(), getType(), etc

Property

• getName(), getType(), etc

Greetings Marc

[mneuhaus]

I've created a little Experiment a few days ago to try your Parser. You can take a look at the example in the readme to get a feeling for what kind of API i'm aiming for:

https://github.com/mneuhaus/CodeBuilder

[nikic]

Hey @mneuhaus! I'm definitely all in favor of adding some more user friendly APIs.

But I don't yet fully understand some things about your particular proposal:
I think many of the getter examples are already implemented as properties. E.g. you can do $method->args or $method->name instead of $method->getArguments() or $method->getName(). (You generally can access all subnodes this way.)
I also didn't quite get what the getByType(Class|Method|Property|...) methods are supposed to do? Where should they go and what do they do?

But I really like the idea of adding simple builder classes for common structures like classes etc. Something like

<?php
$class = $builder->class('SomeLogger')
    ->extend('BaseLogger')
    ->implement('Logger', 'Some\\OtherInterface')
    ->addMethod($builder->method('log')
        ->addParam($builder->param('msg')
        ->addParam($builder->param('type')->optional()->defaultTo('info')
        ->stmts($stmts)
    )
    ->build();

[mneuhaus]

Hey @nikic

great to hear you're open to such additions :) i think this could be benefitial for a lot of frameworks providing kickstarters and such :)

Yea, kind of overlooked the magic getter at first. forget those.

The getByType is basically a shortcut to filter the SubStatements based on a specific type. I've Put this into the NodeAbstract Class:

<?php
    public function getByType($type) {
        $results = array();
        if(is_array($this->stmts)){
            foreach ($this->stmts as $key => $value) {
                if($value->getType() == "Stmt_" . $type){
                    $results[] = $value;
                }
            }
        }
        return $results;
    }
?>

I mainly use this function to implement the getMethod() and getProperties():

<?php
    public function getMethods() {
        return $this->getByType("ClassMethod");
    }

    public function getProperties() {
        return $this->getByType("Property");
    }
?>

I'm currently already working on some parts of this api. i've now chosen to put it into the statements itself instead of a separate class. A Fluent interface would indeed by also quite desirable.

i've even toyed with the idea of a rudimentary way to replace placeholders inside statements. Basically i thought about serializing. That way you could create "Template" classes which contain the needed logic statements inside the functions which you could reuse to add to newly created functions.

Example:

<?php
class Template {
    public function setProperty($___property) {
        $this->___property = $___property;
    }
}
?>

Then you could load this Template and do something like this:

<?php
$template = $builder->parse("Template.php");

$newClass = $builder->class("NewClass");

$method = $template->getMethod("setProperty");
$method->name = "setTitle";
$method->replace("property", "Title");
$newClass->addMethod($method);

$newClass->build();
?>

To get this:

<?php
class NewClass {
    public function setTitle($title) {
        $this->title = $title;
    }
}
?>

To do this replacement i would just serialize the statements, str_replace("___" . $from, $to) and then unserialize.

My head is kind of spinning because of the possibilities with this :D

[nikic]

Sorry for my late response, I kindof forgot about this completely :(

The getByType is basically a shortcut to filter the SubStatements based on a specific type. I've Put this into the NodeAbstract Class:

Ah, okay, now I get it. This seems like a reasonable thing to do.

i've even toyed with the idea of a rudimentary way to replace placeholders inside statements. Basically i thought about serializing. That way you could create "Template" classes which contain the needed logic statements inside the functions which you could reuse to add to newly created functions.

Having stuff like templates is a very interesting idea that could also make a lot of stuff easier. Building the AST up by hand is quite a hassle and leads to ugly code. It would definitely be nice to just define it in terms of real PHP code that just gets some placeholders replaced.

But I think that the implementation of this is much harder than it sounds. At least I doubt that simple serialization will work. PHP's serialization format is not really suitable for search&replace like changes (e.g. strings are saved with a length, which would then be out of sync and unserialize would fail.) This'll need a bit more thought, but I'm sure that there is some elegant solution to this.

[mneuhaus]

Another quite "simple" way to realize templates could be:

1. to load the contents of "Template.php"
2. apply transformations with str_replace
3. parse transformed code

<?php
class Template extends Builder {

    public function __construct($file) {
        $this->content = file_get_contents($file);
    }

    public function getMethod($name, $replacements = array()) {
        $content = $this->content;

        foreach ($replacements as $key => $value) {
            $content = str_replace("__" . $key, $value, $content);
        }
        $ast = $this->parse($content);

        return $ast->getMethod($name);
    }
}

$template = new Template("Template.php");

$newClass = $builder->class("NewClass");

$method = $template->getMethod("setProperty", array("property" => "Title") );
$method->name = "setTitle";
$newClass->addMethod($method);

$newClass->build();
?>

I think such a simple replacement should cover 90%+ of the usecases.

This could be a little performance hit in case of big Templates, but i think perfomance is for such a Builder quite unimportant, since it will probably only be used by developers.

Building the AST up by hand is quite a hassle and leads to ugly code.

I absolutly agree. I think there should only be a api to do structural things, not logic. Classes, Methods, Properties, Namespaces.

[mneuhaus]

I've just coded some initial stuff: Commit: Added Builder, Template and some Usability Methods

• added Builder to parse, render and create classes
• added Template to transform existing PHP code
• added TYPO3CGL PrettyPrinter ( Clone of Zend with a few changes. Properties first then Methods, sort by Alphabet, ...)
• added tests/Builder/Test.php with an example
• added some utility functions to Class, Property, Method and NodeAbstract

Test.php:

<?php

require_once dirname(__FILE__) . '/../bootstrap.php';

$newClass = PHPParser_Builder::createClass("NewClass");


// Get all Statements from the Template and add them to the new Class
$propertyTemplate = new PHPParser_Template(dirname(__FILE__) . "/Templates/PropertyTemplate.php");
$newClass->add($propertyTemplate->getStatements(array("property" => "title")));
$newClass->add($propertyTemplate->getStatements(array("property" => "description")));
$newClass->add($propertyTemplate->getStatements(array("property" => "created")));

// Get a single Method from the Template
$propertyTemplate = new PHPParser_Template(dirname(__FILE__) . "/Templates/MethodsTemplate.php");
$newClass->add($propertyTemplate->getMethod("someNonesense"));

echo "<pre>";
echo htmlspecialchars(PHPParser_Builder::render($newClass));
echo "</pre>";

Result:

<?php 
class NewClass {
    /**
     * My awesome Created
     *
     * @var string
     **/
    protected $created;

    /**
     * My awesome Description
     *
     * @var string
     **/
    protected $description;

    /**
     * My awesome Title
     *
     * @var string
     **/
    protected $title;

    /**
     * Getter for created
     *
     * @return string
     **/
    public function getCreated() {
        return $this->created;
    } 

    /**
     * Getter for description
     *
     * @return string
     **/
    public function getDescription() {
        return $this->description;
    } 

    /**
     * Getter for title
     *
     * @return string
     **/
    public function getTitle() {
        return $this->title;
    } 

    /**
     * Setter for created
     *
     * @param string $created
     **/
    public function setCreated($created) {
        if ($created == 'foo') {
            throw new Exception('Foo isn\'t allowed here...', 1);
        }
        $this->created = $created;
    } 

    /**
     * Setter for description
     *
     * @param string $description
     **/
    public function setDescription($description) {
        if ($description == 'foo') {
            throw new Exception('Foo isn\'t allowed here...', 1);
        }
        $this->description = $description;
    } 

    /**
     * Setter for title
     *
     * @param string $title
     **/
    public function setTitle($title) {
        if ($title == 'foo') {
            throw new Exception('Foo isn\'t allowed here...', 1);
        }
        $this->title = $title;
    } 

    /**
     * Some nonesense Method
     *
     **/
    public function someNonesense() {
        $nonsense = array('foo', 'bar');
        foreach ($nonsense as $key => $value) {
            $nonsense[$key] = str_replace('foo', 'bar', $value);
        }
        return $nonesense;
    } 

}

[nikic]

That looks like a pretty good start for a powerful code generation tool. I'll try to get these ideas into the projects as soon as possible (which should be on this weekend :)

[mneuhaus]

i've just had a idea for a more "fluid" interface:

<?php

$template = new PHPParser_Template(dirname(__FILE__) . "/Templates/PropertiesTemplate.php");

$newClass->add(
    $propertyTemplate->from("StringPropertyClass")
                     ->with(array("property" => "title"))
                     ->getStatements()
);

$newClass->add(
    $propertyTemplate->from("DatetimePropertyClass")
                     ->with(array("property" => "created"))
                     ->getStatements()
);

$newClass->add(
    $propertyTemplate->from("StringPropertyClass")
                     ->with(array("property" => "created")
                     ->getMethod("get__Property")
);

?>

This way you could group several classes in one template and choose from them through "from([classname])".

[mneuhaus]

I've just commited some changes to my fork: Moved most of the Logic to the Builder

Changes:

• Moved most of the Logic to the Builder
• The Builder gets instanciated with a path to a folder containing template files
• added the function "from" to load the appropriate template
• added the function "with" to set the context

Test.php:

<?php

require_once dirname(__FILE__) . '/../bootstrap.php';

$newClass = PHPParser_Builder::createClass("Party");

/**
 * Dir Templates:
 * Templates
 *  - Property
 *     - DateTime.php
 *     - Relation.php
 *     - String.php  
 */
$builder = new PHPParser_Builder(dirname(__FILE__) . "/Templates/");

$newClass->add(
    $builder->from("Property_String")
            ->with(array("property" => "title"))
            ->getStatements());

$newClass->add(
    $builder->from("Property_Relation")
            ->with(array(
                "property" => "guest",
                "properties" => "guests", 
                "class" => "Guest"))
            ->getStatements());

$newClass->add(
    $builder->from("Property_DateTime")
            ->with(array("property" => "date"))
            ->getStatements());

echo "<pre>";
echo htmlspecialchars(PHPParser_Builder::render($newClass));
echo "</pre>";

?>

Here's the result: <?php class Party {...

Some thinks that came additionally to my mind we need to adress:

• existing functions shouldn't replaced. this could possibly be tackled by something like $class->add($stmts, $override = true|false)
• i haven't tested the above with namespaces yet
• maybe it should support multiple classes in one file as well, although this isn't really done in most projects
• more and more frameworks(Doctrine2, FLOW3, Symfony2,...) use the docblocks as configuration areas, so it would be awesome to be able to parse and render them as well. Here's an example from FLOW3:

<?php
/**
 *
 * @FLOW3\Scope("prototype")
 * @Admin\Active
 * @Admin\Group("Testcases")
 * @Admin\Set(title="Default Types", properties="string,integer,float,boolean,date,time,datetime,resource,tag,tags")
 */
class Widgets{
    ...
    /**
     * @var \Doctrine\Common\Collections\Collection<\AdminDemo\Domain\Model\Address>
     * @ORM\ManyToMany(inversedBy="widgets_manytomany")
     * @Admin\Widget("Chosen")
     */
    protected $addressesChosen;
        ...
}

FLOW3 uses the Doctrine2 Parser to parse these statements, the problem is that for that Parser to Work you need appropriate AnnotationClasses like Admin\Annotations[Group|Active|Set|...] which would be very dependend on that library. And it wouldn't provide the opposite direction to render these docblocks again.

I Hope i'm not going overboard with all my crazy ideas :D

[mneuhaus]

Sorry for spamming this issue, but i've just tackled the conflict management as well:

i introduced 4 new constants to define the behavior in case of a conflict:

• CONFLICT_IGNORE
• CONFLICT_REPLACE
• CONFLICT_APPEND
• CONFLICT_PREPEND

<?php
$party->add(
    $builder->from("Property_Relation")
            ->with(array(
                "property" => "guest",
                "properties" => "guests", 
                "class" => "Guest"))
            ->getStatements());

// This call adds all statements problem is, that the __construct already exists
// By Default CONFLICT_IGNORE will ignore this new _construct in favor of the existing
// since we now this construct only has a new initiation in it we specify 
// CONFLICT_APPEND as second argument, to append the Methods logic to 
// methods that already exist
$party->add($builder->from("Property_Relation")
            ->with(array(
                "property" => "beer",
                "properties" => "beers",
                "class" => "Beer"))
            ->getStatements(), PHPParser_Builder::CONFLICT_APPEND);

[nikic]

I started adding code generation features in a separate branch: https://github.com/nikic/PHP-Parser/tree/codeGeneration. Currently it only adds builders. Templating will come next.

Thanks for all your suggestions :)

[nikic]

What I'm thinking about right now is how to make the whole template stuff as general-purpose as possible.

In your above examples the placeholders are just plain text. But there are also uses, where the placeholders are syntax trees themselves. Consider this example (closure emulation):

<?php
class _Closure_%id% extends _Closure {
    public function call(%params%) {
        extract($this->uses, EXTR_REFS);
        %stmts%
    }
}

Here %id% is a plain text placeholder, %params% is a parameter array and %stmts% is a statement array.

I don't think that there is any way to automatically figure out how the the different placeholders should be inserted. Instead, my idea would be to leave the correct insertion up to the template writer:

<?php
class _Closure_{{ $id }} extends _Closure {
    public function call({{ $prettyPrinter->pCommaSeparated($params) }}) {
        extract($this->uses, EXTR_REFS);
        {{ $prettyPrinter->prettyPrint($stmts) }}
    }
}

The whole thing would basically work by compiling the template to PHP (similar to how Twig does it, but with a more trivial approach) and then execute that.

This obviously could work very similarly in your example:

<?php
class GetterSetterTemplate {
    /**
     * @var {{ $type }}
     */
    protected ${{ $name }};

    /**
     * Get {{ $name }}.
     *
     * @return {{ $type }}
     */
    public function get{{ ucfirst($name) }}() {
        return $this->{{ $name }};
    }

    /**
     * Sets {{ $name }}.
     *
     * @param {{ $type }} ${{ $name }}
     */
    public function set{{ ucfirst($name) }}(${{ $name }}) {
        $this->{{ $name }} = ${{ $name }};
    }
}

But as you see in this case this is getting ugly quite quickly. What one could do is provide shortcuts, like the one used in your proposal:

<?php
class GetterSetterTemplate {
    /**
     * @var __type
     */
    protected $__name;

    /**
     * Get __name.
     *
     * @return __type
     */
    public function get__Name() {
        return $this->__name;
    }

    /**
     * Sets __name.
     *
     * @param __type $__name
     */
    public function set__Name($__name) {
        $this->__name = $__name;
    }
}

Here __name would be same as {{ $name }} and __Name same as {{ ucfirst($name) }}.

But I'm not really sure about all that. Maybe I'm just thinking too complicated. What do you think about this?

[mneuhaus]

I also thought about introducing more complex template logic, maybe simply through Twig itself ( no need to reinvent all that power :), but i think this shouldn't be necessary. In my honest opinion i would keep the Templates as simple and syntactically correct as possible. By doing this one could event UnitTest it's Templates easily.

A compromise could be a real simple syntax to allow some filters:

• __name|ucfirst
• __params|toList
• __stmts|prettyPrint

i used the | character as an example here, since it would make the PHP code syntactically incorrect. I'm uncertain which other non Alphanumeric characters beside from "_" are valid in all different Cotexts (MethodNames, ClassName, PropertyNames, Statements etc. )

:)

[mneuhaus]

i'm just reading through this Post: Exotic names for methods, constants, variables and fields - Bug or Feature?

The usage of fancy special characters is strongly discouraged in general, but in our very special case this might be a valid place to use of character that stands out enough among the regular alphabet.

[mneuhaus]

Here's an example, i've used § to encapsulate the variables and ° to apply filters, both of these characters are accessable through most keyboard and valid to the php interpreter:

<?php

class PropertyString {
    /**
     * My awesome §name°ucfirst§
     *
     * @var string
     **/
    protected $§name§;

    /**
     * Getter for §name§
     *
     * @return string
     **/
    public function get§name°ucfirst§() {
        return $this->§name§;
    }

    /**
     * Setter for §name°ucfirst§
     *
     * @param string $§name§
     **/
    public function set§name°ucfirst§($§name§) {
        $this->§name§ = $§name§;
    }
}

new PropertyString();
?>