[Form] RFC: Integrate Zend\Filter into Form

[webmozart]

People have voiced the need for form input filtering various times. Filtering could be added to the Form component very easily by creating an (optional) extension that enables the following API:

$builder->add('content', 'textarea', array(
    // a single filter or an array of filters
    'filters' => new StripTags(),
));

The big question is which library to use for filtering. Upon short research, I came upon two existing alternatives:

• Zend\Filter
  • appears mature
  • potentially brings in other ZF2 baggage, which I don't care about once ZF2 supports Composer properly and unless the baggage creates further disadvantages
• DMS-Filter
  • very young
  • supports annotation-based filtering in POPOs

Does anyone know further filtering libraries? Does anyone have experience with Zend\Filter in non-ZF(2) projects?

Comparison Sheet

Zend\Filter	DMS-Filter	Comment
Alnum	Alnum	both support Unicode, Zend has special treatment for Japanese, Chinese and Korean
Alpha
Boolean	Boolean	DMS does a simple cast, Zend's implementation is more complicated, but supports localized "yes" and "no" as well as leaving the original value unchanged unless it evaluates to false (use case?)
Callback
Digits
File\LowerCase
File\Rename
File\UpperCase
	Float
HtmlEntities
Int	Int
PregReplace		supports Unicode
StringToLower	ToLower	both support Unicode
StringToUpper	ToUpper	both support Unicode
StringTrim	Trim	both allow the specification of a character list to be trimmed
StripNewlines	StripNewlines
StripTags	StripTags	Zend allows to specify allowed tags, attributes and comments; DMS only allowed tags (using `strip_tags` internally).
Not a filter, but a transformer
BaseName
Compress		supports BZ2, GZ, LZF, RAR, TAR and ZIP
Decompress		see Compress
Decrypt		supports MCrypt and OpenSSL
Dir
Encrypt		see Decrypt
File\Decrypt		see Decrypt
File\Encrypt		see Decrypt
Null
RealPath
Word\CamelCaseToDash
Word\CamelCaseToSeparator
Word\CamelCaseToUnderscore
Word\DashToCamelCase
Word\DashToSeparator
Word\DashToUnderscore
Word\SeparatorToCamelCase
Word\SeparatorToDash
Word\SeparatorToSeparator
Word\UnderscoreToCamelCase
Word\UnderscoreToDash
Word\UnderscoreToSeparator

[daFish]

That would be a great addition to the form component. The only drawback I see when opting for ZF2 would be the huge list of dependencies on other ZF2 components.

[rdohms]

Also, DMS\Filter can have many more filters, just make a request for them and i'll get on that. I already have a bundle for it and would be glad to fix anything left behind.

Thanks @bschussek

[marijn]

I've got a few questions about filters:

• will they be applied before or after transformations?
• will they be bound to the form layer or the validation layer?
• are they unidirectional or bidirectional?
• does it convert text to lowercase or does it filter the non lowercase values?

[immutef]

👍 for DMS-Filters. It already makes use of Doctrine\Common (which is, in fact, a Symfony2 dependency) and I love the fact that Symfony2 uses/supports small stand-alone-one-task-at-a-time libraries (such like Monolog).

[webmozart]

@marijn Thanks for bringing this up. Filters would use the BIND_NORM_DATA event, i.e. they would be applied on the normalized data during binding. The normalized data would then be transformed back to the client and application format in order to reflect the changes of the filter.

Filters are unidirectional. Bidirectional filters are no filters, but data transformers.

What the filter does depends entirely on its implementation. StripTags for example strips tags away, StringToLower converts to lowercase characters etc.

@rdohms Thanks for the feedback. I will keep it in mind.

[antoinegomez]

Not sure that Zend\Filter has so many dependencies, using a more mature and complete solution is better.

ZF2 will use some SF2 features, so why can't we do the same ? :)

[weierophinney]

I'll clarify a few points about Zend_Filter. ZF2's filters are primarily oriented at normalization -- not validation. As such, they generally transform the input and return the transformed version -- thus "StringToLower" transforms the input to lowercase. Zend\Filter\FilterChain can be of interest for chaining multiple filters, and thus creating an end result that is the product of each filter -- a filter with StringTrim followed by Alpha followed by StringToLower would transform " the WORD" to "theword".

Regarding dependencies -- we plan to go through the package and identify optional dependencies as distinct from required dependencies. In most cases, the only dependencies are optional -- they're necessary if you use a particular filter, but not another. In some cases, dependencies will be disappearing; as an example, because we're rewriting the i18n/l10n layer, and the approach will be quite different, many of the locale-aware filters will either remove that dependency, be moved to a separate component, or make the dependency optional. This work should be largely complete by our upcoming beta4 release (scheduled sometime in the next 4 weeks).

One additional note, Zend\Filter\Input will be removed for beta4, as it is being replaced by a new Zend\InputFilter component (which bridges Zend\Filter and Zend\Validator).

[ericclemmons]

I've been using Zend\Filter successfully for exactly this purpose in both Zend Framework and my Symfony2 projects. The only requirement would be, as @weierophinney mentioned, remove or identify any dependencies and get those straightened out for befa4.

[lsmith77]

@weierophinney indeed .. all we are missing is a solution for normalization/sanitization

DMS filter has the advantage of using similar infrastructure than what we already use in Symfony2 (at least for annotations).

Looking at simply the number of filters provided by https://github.com/rdohms/DMS-Filter/tree/master/Rules and https://github.com/zendframework/zf2/tree/master/library/Zend/Filter its clear that Zend Filter is more mature and I guess if they do refactor it to Zend\InputFilter then at least the broad set of filters will remain and I doubt that there will much instability introduced in the process.

That being said, I don't think its very hard to get DMS up to speed, especially since Zend Framework is BSD licensed, we could in theory just take their filters and adapt them into DMS.

[rdohms]

@lsmith77 indeed. DMS\Filter was modeled after Symfony\Validation so yes it does really look similar.

I did take a look at the ZF Filter list and discussed this with @weierophinney and did indeed note that we follow very different paths. I have a focus on annotations and pure unidirectional filtering, while ZF has filtering/transformation. Since Symfony already has transformation i really focused on the missing filtering step.

Please feel free to open any issues with Filter requests, I will be implementing a few hooks and rules and generally updating the library in the next months.

[webmozart]

@weierophinney Thank you for the clarification. Do you have any plans (and a schedule) to add Composer support to the individual components?

In general, I don't think that the surrounding infrastructure of the filters (annotations, filter chains etc.) is of any importance for us. We just need a set of filters that transform values into filtered values. These filters will be executed sequentially directly by the form extension. I especially don't want to mix the data model with input filtering, i.e., put filter rules into the model via annotations. Doing this violates SoC.

Objectively speaking, Zend\Filter seems far more mature (multibyte support, encryption, compression) and thus is my clear favorite. If it manages to

• reduce dependencies to a minimum
• add composer support

it fulfills all our requirements.

@rdohms I appreciate your offer, but I don't see the point in duplicating efforts unless it makes a big difference for us.

[rdohms]

@bschussek No prob. Just exaplin to me why annotation based filtering violates the SoC and not validation? To me they are 2 parts of the same coin. Having filtering on form level does not also eliminate the need for entity level filtering.

[stof]

@rdohms The validator component is about validating the object, not the form. On the other hand, the filtering described by @bschussek previously would be applied on the normData of the form, not on the appData (i.e. the object), so it is part of the form level, not of the domain object level

[daFish]

@stof But that would leave out filtering of data for scenarios were no form is used with an domain object. This might be necessary and useful.

[stof]

@daFish if you have some stuff needing to be done each time you set a value in an object, the right place to put the logic is the setter.

[daFish]

@stof The same argument can be applied to validation: Everytime I set a value on my object it should be validated.

[Seldaek]

@stof what about DRY? Some filters have a lot of code, putting such code in every setter is quite a recipe for disaster.

[rdohms]

i have to agree with @daFish the motivation behind filtering in annotations is the same as for validation. So if validation is auto executed by forms, filtering to me is on the same level.

Validation in Symfony forms is not available on the form configuration IIRC, so why would filtering be form specific and not part of the domain?

[marijn]

Thus far I agree with @daFish on this.

If Zend\Filter also does transformations than that's an argument against using it. I imagine a lot of newcomers that are familiar with Zend\Filter trying to use it for transformation (which is not the intended use in this implementation context).

[webmozart]

We could support filtering both on domain objects and on forms directly (like with validation constraints, although this is only partially finished by now).

@rdohms: What about basing DMS-Filters on Zend\Filter instead of implementing your own rules? This way we could have the best of both worlds.

[rdohms]

@bschussek i had this talk with @weierophinney yesterday. What is your definition of basing them on ZF? Adding ZF as a depedency? or defining the same filters?

I was actually looking into ZF yesterday to separate data transformations from filtering, and most of the actual filters are already implemented in DMS, but i'always open to suggestions and improvements, i have been meaning to leverage the filter_* functions in the filters, whenever it makes sense.

[webmozart]

@rdohms I'm speaking about actually using their filters, i.e. anything that implements Zend\Filter\Filter, only from within your walker.

[rdohms]

@bschussek i have a little bit of an issue with that, as the library was meant to have a small footprint. I can however look into making a hook so it can plug ZF in maybe. Will look into the interfaces and see if that possible without creating frankenstein,

[guilhermeblanco]

Hi!

I'm using DMS-Filter and I'm pretty happy with its content.
It seems it's only missing 3-5 filters to have the same functionality as Zend\Filter and I after suffering from Zend dependencies for over 2 weeks, I'm more likely to say DMS-Filter is light years easier to be used than actually Zend\Filter.

I'd also attempt as much as I could to bring 20+ dependencies only to be able to have a filtering librabry.
That's exactly why I prefer a lightweight solution and it seems DMS-Filter fits perfectly into this.

Cheers,

[immutef]

@weierophinney is there an estimation when a first RC is planned or when we could expect a stable ZF2 release?

[webmozart]

@guilhermeblanco:

I'm more likely to say DMS-Filter is light years easier to be used than actually Zend\Filter.

Can you elaborate?

It seems it's only missing 3-5 filters to have the same functionality as Zend\Filter

This is wrong. Please see the comparison sheet I added above.

[jalliot]

@bschussek Some of the filters supported by Zend don't make sense for the Form component IMHO (like (Un)Compress, (De)Encrypt f.e. because they have to be bidirectional, all filters related to files (Basename, Realpath, etc.)).
So your comparison sheet should maybe include only what would be of some use for us.

[webmozart]

@jalliot you are right about compression and decryption. The file filters on the other hand do well make sense for uploaded files. This way, an uploaded file could be automatically en/decrypted or (de-)compressed. I like this.

Having the compression and encryption filters decoupled from the file specific ones does not hurt, so I won't strike them out above.

[jalliot]

I didn't thought of that... That could be indeed pretty cool if handled correctly

[rdohms]

@bschussek it would be nice to flag what is a filter and what is actually data transformation. Basename for example I consider to be data transformation not filtering. Same with the compress/encrypt stuff. Filtering has a different purpose of avoiding bad data to come through, not of auto-converting data into other formats.

Even Boolean is on a thin line here between transformation and filtering, in my approach i just did a simple cast so that its interpreted as a boolean after filtering.

If we are looking for transformation and filtering in this implementation, then DMS has no way of competing.

[webmozart]

@rdohms You raise a valid point. I adjusted the table accordingly.

[rdohms]

@bschussek nice, ok I already did PregReplace this week, have not tagged a new release yet, but i'll take a look into that list and implement a few more. To your point of using ZF in DMS, i'm going to see if its possible to implement a ZendRule, so it would be something like:

@filter\Zend('Zend\Filter[name]', [params])

That way it can leverage the Zend filters if you have them locally. Have not looked if that's possible, but if its interesting to anyone i'll work on it.

[webmozart]

To your point of using ZF in DMS, i'm going to see if its possible to implement a ZendRule, so it would be something like:

That's not quite what I meant and probably a waste of time. I was rather speaking about avoiding to duplicate all the filter implementations (and their maintenance).

[rdohms]

@bschussek i know, but i would rather keep a functional DMS library with reasonable support and leverage extra filters without requiring them. That way people like @guilhermeblanco can still use it in a simple format, and people who want to leverage ZF can also do it.

I seriously do not want to make the library depend on ZF, so i would like to keep it optional. Even if its not "selected" for this feature, feels like i'm betraying the reason i made it in the first place. But yeah that a personal opinion.

[guilhermeblanco]

@bschussek My idea is pretty simple. When I started to use Zend\Filter, I remember that I had to import around 10 dependencies to get it working isolated.
Then I figured it out I had to write a Service class that allows filtering of not only form related VOs, but also non-form related VOs. That was the exact point I decided to switch to DMS-Filter.

When I made the switch to DMS-Filter, all I had to do is add a simple EventSubscriber (I have to admit it's a very dirty code I wrote in 5min) and add a new instance on every form type.

Of course that would be incredibly easier if form component includes a new event monitoring through a tag. That would simplify my life.
I can even foresee a possible implementation that DMSFilterBundle could do taking advantage of this functionality:

<service id="dms.filter.listener.filter" class="DMS\FilterBundle\Listener\FilterSubscriber">
    <argument type="service" id="dms.filter.service.filter" />
    <tag name="form.event_listener" event="postBind" method="postBind" />
</service>

By doing that, it means Subscriber would be always active, so I wouldn't need to keep instantiating it every time I have a Form Type.

In the case of using the component standalone (for non-form VOs), I imply need to grab the "dms.filter" service.
I can't see a cleaner solution than this enhancement to Form component on Symfony. I'm pretty comfortable with the state of DMS-Filter ATM and I may need for one or two filters in the future.

As people already mentioned, I used it only for input filtering, not for transformation. That can be a good reason why the tool fits so well on my specific environment.

[stof]

@guilhermeblanco forms are not managed by the DIC so using a tag would not work (the DIC cannot hook in your query builder). However, if you want to have a subscriber which is always active, you could create a form type extension applied on form which adds the subscriber for the root (and this extension can be registered through a tag).

[guilhermeblanco]

@stof I don't see how it can't be managed.
There're tons of manageable tags that hooks form types, forms and etc already. It doesn't seem that DI depends on Forms, so why not make Form package depending on DI? It doesn't generate any cyclic dependency from my point of view

[stof]

@guilhermeblanco The issue is that you don't have one Form but many (even if your controller only use one form, there is many Form objects as each subform is a Form too). So on which one would you register the listener (each one has its own dispatcher). Tags are allowing to hook into the factory, which is a container-scoped service. I don't see how you could use tags for the forms (keep in mind they are only available at compile-time, not at runtime in the container)

[AurelC2G]

Any news about this?

[stof]

@AurelC2G as we are in beta now, we won't add any new features for 2.1 but only fix bugs. New features will be for the 2.2 release now.

[AurelC2G]

I know and I was not expecting anything sooner than 2.2, just wondering whether this subject had been completely forgotten ;)

[rdohms]

@AurelC2G DMS\Filter is now automatically hooking into Symfony Form using the DMSFilterBundle, so its a easy drop in for 2.1 until this is finalized.

[stof]

@AurelC2G it is not forgotten. It is delayed as the priority is the 2.1 stable release currently, not the 2.2 features

[ghost]

bump

[vladcosorg]

Hey! Is there any progress on this issue? I suppose it won't be in 2.3 either? :(

[acasademont]

Hi there! This would be nice indeed :)

[rdohms]

dms/dms-filter 2.0 will come with support for embedding Zend\Filter, add to that the DMSFilterBundle and it can auto-filter entities submitted by forms as well as work as described above.

Both 2.0 releases should come out next week, the filter is done the bundle just needs wrapping up to support Symfony 2.3.

I'm open to feedback on anything that's missing and hope that this gives you all a good option to filtering in Symfony.

[gfreeau]

Symfony needs input filtering to go along with validation.

@rdohms it would be great for DMSFilerBundle to support all standard config formats including yml and xml.

[gfreeau]

I added xml/yml support to DMS Filter

rdohms/dms-filter#10
rdohms/dms-filter#9

I plan to add support to the sf2 bundle as well.

I noticed there is a lot of cross over between the validator component and this filter component such as loading xml and yml config and metadata.

I think if symfony2 is going to have a filter component some re-factoring could be done to share some code between the two.

[javiereguiluz]

It's been more than 2 years since this discussion was initiated. Should we definitely close it as won't fix or should we officially add it to the roadmap of the next Symfony version?