Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Schema Validation for Joomla 2.5/3.1 Extensions #838

Closed
cedricwalter opened this issue Mar 18, 2013 · 28 comments
Closed

Schema Validation for Joomla 2.5/3.1 Extensions #838

cedricwalter opened this issue Mar 18, 2013 · 28 comments
Labels

Comments

@cedricwalter
Copy link

@cedricwalter cedricwalter commented Mar 18, 2013

I’ve created from scratch, all Joomla extensions XSD at

https://github.com/cedricwalter/joomla-xsd

Without them, Joomla accept any entry in manifest xml and never complains about

  • Mistyping, like a valid xml but that the Joomla installer do not understand or only partially,
  • Wrong constructs, xml tag child misplaced,
  • Invalid data type, like a path not being a valid path, an expected integer being a text and so on…

Joomla just silently die during install or install only partially extensions.

These days are over as developers with any decent IDE will be able to enjoy

  • validation (syntax and values)
  • auto completion (suggestion)
    while typing!,

Ideally these files should be available online on jooomla.org domain

Something like
http://www.joomla.org/xds/2.5/plugins.xsd
http://www.joomla.org/xds/2.5/modules.xsd

Cédric

@chrisdavenport
Copy link
Contributor

@chrisdavenport chrisdavenport commented Mar 18, 2013

Looks good. Location should perhaps be http://developer.joomla.org/schemas/2.5/plugins.xsd ?

Chris.

@cedricwalter
Copy link
Author

@cedricwalter cedricwalter commented Mar 18, 2013

Yes look better to me as well. We should find something and keep it fot the
next xx years to come.

Regards
Cédric
On Mar 18, 2013 9:04 PM, "Chris Davenport" notifications@github.com wrote:

Looks good. Location should perhaps be
http://developer.joomla.org/schemas/2.5/plugins.xsd ?

Chris.


Reply to this email directly or view it on GitHubhttps://github.com//issues/838#issuecomment-15078193
.

@nicksavov
Copy link
Contributor

@nicksavov nicksavov commented Mar 22, 2013

Thanks for coding this, Cédric! This is awesome! :)

While we’re transitioning to a new integrated tracker, could you post this on our current main tracker at JoomlaCode and cross-reference each with a link to other? http://joomlacode.org/gf/project/joomla/tracker/?action=TrackerItemBrowse&tracker_id=8549

Alternatively, let me know if you’d like me to create it for you and I can go ahead and do that.

Thanks in advance and thanks again for coding this, Cédric!

@cedricwalter
Copy link
Author

@cedricwalter cedricwalter commented Mar 22, 2013

Hi Nick

Thanks, i was badly needing this as i was searching for an installer bug in one of my extension. Took like 5 hours to do, tested against 80% of joomla 2.5 manifest.

I will go to joomla 3.0 soon and update in a branch all schema if needed.

tracker cross post http://joomlacode.org/gf/project/joomla/tracker/?action=TrackerItemEdit&tracker_item_id=30358

@cedricwalter
Copy link
Author

@cedricwalter cedricwalter commented Jun 10, 2013

Hi All

Any news/status on this pull request?

@dongilbert
Copy link
Contributor

@dongilbert dongilbert commented Aug 16, 2013

This is awesome. I'm all for this. Can you update this for 3.1? There shouldn't be too many changes.

We're all excited to see you do this.

@cedricwalter
Copy link
Author

@cedricwalter cedricwalter commented Aug 17, 2013

Yes I will update this to 3.1, and start also tagging (to keep 2.5 compatibility)

@elkuku
Copy link
Contributor

@elkuku elkuku commented Aug 17, 2013

Big 👍

The Joomla! project is really lucky to have contributors like @cedricwalter - kudos

Please read also: Are we taking care of our own?

@dongilbert
Copy link
Contributor

@dongilbert dongilbert commented Aug 17, 2013

Definitely - I'm really sad this has been sitting around for 5 months and I wasn't aware because it was on the CMS side of things. Hopefully we start doing better on recognizing contributors like this.

@cedricwalter
Copy link
Author

@cedricwalter cedricwalter commented Aug 18, 2013

Is there any better documentation than http://docs.joomla.org/Manifest_files ?

or should i like last time try to validate all existing manifests (packages, components, modules, library and plugins) of Joomla 3.1.5?

This way i will see if there is any new joomla 3.1 manifest features.... it just take some time but we are then on the safe side....

What do you think?

@cedricwalter
Copy link
Author

@cedricwalter cedricwalter commented Aug 19, 2013

Full support for joomla 3.1, tested with all 113 manifests of joomla 3.1.5! (components, modules, plugins and templates)

component / module / plugins

  • add in <menu> support for attribute value img="class:banners">
  • add in <menu> support for attribute value view="anyString"
  • add in <menu> support for attribute value alt="anyString"
  • add to type="cmsVersionType" version 3.1
  • add support for <menu link="option=com_finder">COM_FINDER</menu> (finder.xml)
  • In <extension> attribute valuemethod="" is now optional
  • add new <help key="ANY_STRING" />
  • add in <field> support for attribute value first="anyNumber"
  • add in <field> support for attribute value last="anyNumber"
  • add in <field> support for attribute value step="anyNumber"
  • add in <field> support for attribute value published="" (mod_articles_category.xml)
  • add in <field> support for attribute value format="%Y-%m-%d %H:%M:%S"
  • add in <field> support for attribute value disable="separator" (mod_login.xml)
  • add in <fields> support for attribute value addfieldpath="validPath" (mod_finder.xml)
  • in <field> validate css class names class="btn-group" or class="btn-group btn1 blue" or class=""
  • Allow empty without fieldset (vote.xml)
    <fields name="params"> </fields>
  • In <authorEmail> consider N/A as a valid email
  • Attribute 'label' now optional on element 'field'.
    <field name="spacer3" type="spacer" hr="true" />
  • Support for validate <field> type="url" (sef.xml)
  • in <fieldset> allow Attribute value label="" to appear in element (debug.xml)
  • allow <field> to have type attribute value category (contactcreator.xml)
  • allow <field> to have new attribute extension=com_* (contactcreator.xml)
  • In <media> attribute destination value is now optional
  • In <fieldset> add optionaladdfieldpath="" and validate that it is a valid path
  • In ```<option value="" empty values are now allowed
  • <updateservers> is now available in plugins manifests
  • In <field> type now support type="modal_article" or from enum (using xsd union)

Plugins only

  • in <file plugin="weblinks">weblinks.php</file>

Template only

  • In <extension> attribute value method="" is now optionnal
  • In <extension> addattribute value client=""
  • add <languages></languages>
@dongilbert
Copy link
Contributor

@dongilbert dongilbert commented Aug 19, 2013

That would be the best written documentation, aside from the actual manifests themselves. Thanks again for putting so much into this. Greatly appreciated. I owe you a drink at JWC if we both make it out there.

@cedricwalter
Copy link
Author

@cedricwalter cedricwalter commented Aug 19, 2013

I've never drank alcohol, but a coke will do just fine ;-)

BTW i still need to add in a lot of places the inline xs:documentation
Using http://docs.joomla.org/Manifest_files do not explain all the stuff found after all the reverse engineering done with all joomla 3.1.5 files...

<xs:annotation>
            <xs:documentation xml:lang="en">xxxxxx</xs:documentation>
</xs:annotation>

It is a good start and should cover more than 99% of all manifests (i found a lot of errors in my extensions thanks to these XSDs )

@mbabker
Copy link
Contributor

@mbabker mbabker commented Sep 12, 2013

Just some notes as I do a quick review:

  • The <installfile> and <uninstallfile> tags in the component manifest are dropped at 3.0
  • Packages support the <scriptfile> tag since 2.5, they also support the <updateserver> tag
  • I'm not familiar with how the validation will work, but will an error be thrown if you use a group element for plugin manifests if it isn't in the list?
@cedricwalter
Copy link
Author

@cedricwalter cedricwalter commented Sep 12, 2013

All Done and more in c11ffdeb1d7d91803375034847066c49ab32e329

  • Remove and tags in the component manifest (dropped at 3.0)
  • Packages now support the and tag
  • In field the attribute class now support list of css classes class="btn-group .blue .bigger">
  • Add more inline documentation
@cedricwalter
Copy link
Author

@cedricwalter cedricwalter commented Feb 14, 2014

HI all

Any news or progress on this ticket?

Thanks

@vdespa
Copy link
Contributor

@vdespa vdespa commented Sep 13, 2014

Thanks for your work so far.

If this still in sync with Joomla 3.3?

This comment was created with the J!Tracker Application at http://issues.joomla.org/.

@vdespa
Copy link
Contributor

@vdespa vdespa commented Sep 13, 2014

Maybe this should be open sourced at github.com/joomla-projects?

This comment was created with the J!Tracker Application at http://issues.joomla.org/.

@cedricwalter
Copy link
Author

@cedricwalter cedricwalter commented Sep 16, 2014

Hi

I would like to maintain it actively, but if nobody use it i don't see the point. I've made this contribution more than a year ago and it is still not used. Can any Joomla team members update me first if they want this?

@Bakual
Copy link
Contributor

@Bakual Bakual commented Sep 16, 2014

I think it would be worth it, and several PLT members (Michael, Nick, Chris) also were in favor of it from reading over the discussion.
Is there something left that needs to be done beside moving it to Joomla property (Joomla repository and domain) and updating all XML files to point to this spec?
Is there a way to test this?

@cedricwalter
Copy link
Author

@cedricwalter cedricwalter commented Sep 16, 2014

I could write a small junit test that load all manifests recursively of Joomla, update every xml and try to validate them against the specs. BTW this is how i did all XSDs, i did load all manifest in a XML editor (oxygenXM)L, and tried to reverse engineer all possible combinations by looking at "what was done in the wild". Some fields validations are evident like email, date, file, directory and so on some are not. I just need more sample to exercise the XSDs. I found them able to validate any joomla 3.3 manifest some months ago. So it is for me features complete.

I did look at joomla online documentation for the xs:documentation</xs:documentation but there also it is not complete. (Only English is done for now)

This is clearly a work in progress that need maybe also to provide branches for joomla 2.5/3.0? till 3.3?
In the beginning i could imagine Joomla would only log the issues found in manifests.
While at a later point, Joomla installer should simply refuse installing invalid manifests.

@betweenbrain
Copy link
Contributor

@betweenbrain betweenbrain commented Sep 16, 2014

👍

@sovainfo
Copy link
Contributor

@sovainfo sovainfo commented Sep 16, 2014

Would like to see Joomla to reject invalid manifests as of yesterday. Only manifests that we know are valid should be updated so they can and will be validated upon install. Ofcourse that requires XSD's to be made available. No need to change all xml files in one go.

@vdespa
Copy link
Contributor

@vdespa vdespa commented Sep 17, 2014

This can be a long discussion. @Bakual can you help (or forward for discussion) by creating a space on the Joomla project so that that @cedricwalter can make a PR or commit the proposed files.

Additionally gathering valid manifest files and creating some tests would be the next step.

It would be a pity to lose such a valuable contribution.

@Bakual
Copy link
Contributor

@Bakual Bakual commented Oct 31, 2014

@cedricwalter I've created a new repo https://github.com/joomla/schemas where we can move those schemas in. The idea is that this repo then would also hold other schemas if there are some. Also it likely needs some sort of versioning.
So it needs a directory structure like /xsd/v3/schemafiles_go_here. What do you think?

When the repo is ready with the data, we thought to copy the files to schemas.joomla.org. The schema definition link in the XML files would then become http://schemas.joomla.org/xsd/v3/component.xsd

Does that make sense?
If so, can you do a PR against that repo?

@cedricwalter
Copy link
Author

@cedricwalter cedricwalter commented Nov 1, 2014

Hi All,

I did create a pull request (joomla/schemas#1) and
also a unit test that load all Joomla manifest and run a validate for each
file found

so far i get only a few validations errors against Joomla 3.3.6 :-)

I am now checking if they are real manifests errors or if schemas required
some changes


Error 1871: Element 'params': This element is not expected. in file:///C:/Users/cedric/Documents/galaxiis/www/dev3/components/com_mailto/mailto.xml on line 29
Error 1845: Element 'mosparam': No matching global declaration available for the validation root. in file:///C:/Users/cedric/Documents/galaxiis/www/dev3/components/com_mailto/views/sent/metadata.xml on line 2
Error 1871: Element 'customContent': This element is not expected. in file:///C:/Users/cedric/Documents/galaxiis/www/dev3/administrator/modules/mod_custom/mod_custom.xml on line 16
Error 1871: Element 'customContent': This element is not expected. in file:///C:/Users/cedric/Documents/galaxiis/www/dev3/modules/mod_custom/mod_custom.xml on line 16
Error 1871: Element 'fieldset': This element is not expected. Expected is ( fields ). in file:///C:/Users/cedric/Documents/galaxiis/www/dev3/modules/mod_languages/mod_languages.xml on line 29
Error 1871: Element 'field': This element is not expected. Expected is ( fieldset ). in file:///C:/Users/cedric/Documents/galaxiis/www/dev3/plugins/system/languagecode/languagecode.xml on line 24

Cédric

On Fri, Oct 31, 2014 at 8:43 PM, Thomas Hunziker notifications@github.com
wrote:

@cedricwalter https://github.com/cedricwalter I've created a new repo
https://github.com/joomla/schemas where we can move those schemas in. The
idea is that this repo then would also hold other schemas if there are
some. Also it likely needs some sort of versioning.
So it needs a directory structure like /xsd/v3/schemafiles_go_here. What
do you think?

When the repo is ready with the data, we thought to copy the files to
schemas.joomla.org. The schema definition link in the XML files would
then become http://schemas.joomla.org/xsd/v3/component.xsd

Does that make sense?
If so, can you do a PR against that repo?


Reply to this email directly or view it on GitHub
#838 (comment).

Regards,
Cédric

www.galaxiis.com
www.waltercedric.com
www.cedricwalter.com

@betweenbrain
Copy link
Contributor

@betweenbrain betweenbrain commented Dec 24, 2014

@cedricwalter please note that customContent is a valid, allbeit seemingly undocumented, element. Please see

and
$hasContent = empty($this->item->module) || isset($this->item->xml->customContent);
for its usage.

@cedricwalter
Copy link
Author

@cedricwalter cedricwalter commented Dec 30, 2014

@betweenbrain it is already corrected see https://github.com/joomla/schemas/blob/master/xsd/v3/module.xsd#L30

in fact i came across this hidden feature while running the unit test against all .xml files of the latest joomla 3.3.x

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet