xsd2py failing

[thomas-mangin]

Hello Felix,

I never had to use xsd/wsdl/soap before but I am trying to find a library which can help me implement correctly a tr-069 server (instead of just generating/parsing the XML 'by hand') like many available open source projects.

soapfish seems to be a good contender for the job but before I can even get started I have this issue.
Any help would be welcomed ! Thank you.

# xsd2py https://www.broadband-forum.org/cwmp/cwmp-1-4.xsd
Traceback (most recent call last):
  File "/usr/local/bin/xsd2py", line 9, in <module>
    load_entry_point('soapfish', 'console_scripts', 'xsd2py')()
  File "/Users/thomas/source/tr-69/soapfish/soapfish/xsd2py.py", line 171, in main
    code = generate_code_from_xsd(xmlelement, encoding='utf-8')
  File "/Users/thomas/source/tr-69/soapfish/soapfish/xsd2py.py", line 69, in generate_code_from_xsd
    standalone=True)
  File "/Users/thomas/source/tr-69/soapfish/soapfish/xsd2py.py", line 151, in schema_to_py
    return tpl.render(schema=schema, cwd=cwd, base_path=base_path)
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 989, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 754, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/thomas/source/tr-69/soapfish/soapfish/templates/xsd", line 11, in top-level template code
    {{- resolve_import(i, known_files, schema.targetNamespace, cwd, base_path) }}
  File "/Users/thomas/source/tr-69/soapfish/soapfish/xsd2py.py", line 48, in resolve_import
    location = os.path.relpath(location, base_path)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 430, in relpath
    start_list = [x for x in abspath(start).split(sep) if x]
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 360, in abspath
    if not isabs(path):
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 54, in isabs
    return s.startswith('/')
AttributeError: 'NoneType' object has no attribute 'startswith'

[ngnpope]

I have been working on a lot of fixes and this one might be solved in my improvements branch by the commit 828e4a5. I still have a few more things to finish before I submit a pull request.

According to the specification xsd:import[schemaLocation] and xsd:include[schemaLocation] are optional. Please try the branch to see if you have any success. There are also some other issues with generation of names for schemas based on location that I am currently attempting to fix.

[FelixSchwarz]

@thomas-mangin Can you try Nick's branch as suggest in his comment above?

[thomas-mangin]

I was pretty sure to run this branch but to make absolutely sure, I done this ( OSX )

# git branch -v
* improvements 58e7fc3 misc: drop support for django<1.6.0
  master       2407838 merge element reordering for correct XSD restriction/extension rendering (by Iuri de Silvio)

# git log | head -8
commit 58e7fc39e961ff7ccb644e36f5e3024345ba6eb8
Author: Nick Pope <nick.pope@flightdataservices.com>
Date:   Mon Mar 21 16:32:36 2016 +0000

    misc: drop support for django<1.6.0

    License: BSD

# pip uninstall soapfish
Uninstalling soapfish-0.6.0.dev0:
  /usr/local/lib/python2.7/site-packages/soapfish-0.6.0.dev0-py2.7.egg
Proceed (y/n)? y
  Successfully uninstalled soapfish-0.6.0.dev0

# cd  /usr/local/bin
# rm xsd2py wsdl2py py2xsd py2wsdl
# cd -

# python setup.py install
[...]
Using /usr/local/lib/python2.7/site-packages
Finished processing dependencies for soapfish==0.6.0.dev0

# find /usr/local/lib/python* | grep soapfish | cut -f1-7 -d/ | uniq
/usr/local/lib/python2.7/site-packages/soapfish-0.6.0.dev0-py2.7.egg

# find /usr/local/opt/python* | grep soapfish

# which xsd2py
/usr/local/bin/xsd2py

# cat /usr/local/bin/xsd2py
#!/usr/local/opt/python/bin/python2.7
# EASY-INSTALL-ENTRY-SCRIPT: 'soapfish==0.6.0.dev0','console_scripts','xsd2py'
__requires__ = 'soapfish==0.6.0.dev0'
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.exit(
        load_entry_point('soapfish==0.6.0.dev0', 'console_scripts', 'xsd2py')()
    )

/usr/local/bin/xsd2py https://www.broadband-forum.org/cwmp/cwmp-1-4.xsd
Traceback (most recent call last):
  File "/usr/local/bin/xsd2py", line 9, in <module>
    load_entry_point('soapfish==0.6.0.dev0', 'console_scripts', 'xsd2py')()
  File "/usr/local/lib/python2.7/site-packages/soapfish-0.6.0.dev0-py2.7.egg/soapfish/xsd2py.py", line 171, in main
    code = generate_code_from_xsd(xmlelement, encoding='utf-8')
  File "/usr/local/lib/python2.7/site-packages/soapfish-0.6.0.dev0-py2.7.egg/soapfish/xsd2py.py", line 69, in generate_code_from_xsd
    standalone=True)
  File "/usr/local/lib/python2.7/site-packages/soapfish-0.6.0.dev0-py2.7.egg/soapfish/xsd2py.py", line 151, in schema_to_py
    return tpl.render(schema=schema, cwd=cwd, base_path=base_path)
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 989, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/local/lib/python2.7/site-packages/jinja2/environment.py", line 754, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python2.7/site-packages/soapfish-0.6.0.dev0-py2.7.egg/soapfish/templates/xsd", line 11, in top-level template code
    {{- resolve_import(i, known_files, schema.targetNamespace, cwd, base_path) }}
  File "/usr/local/lib/python2.7/site-packages/soapfish-0.6.0.dev0-py2.7.egg/soapfish/xsd2py.py", line 48, in resolve_import
    location = os.path.relpath(location, base_path)
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 430, in relpath
    start_list = [x for x in abspath(start).split(sep) if x]
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 360, in abspath
    if not isabs(path):
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py", line 54, in isabs
    return s.startswith('/')
AttributeError: 'NoneType' object has no attribute 'startswith'

So it seems the bug is still present in this branch -- sorry 😢

[thomas-mangin]

And sorry - I should have said I was running Nick's branch - but looking at the recent tickets it looked like the code to run to try parse complex XSDs.

[FelixSchwarz]

Thank you for your quick response. Unfortunately I'm short on time (easter around the corner) so if you need something faster than 2 weeks you need either to debug this yourself or look at another library...

attached reproducer file so it can't go away: cwmp14.txt

[thomas-mangin]

Being an open source developer too, I can appreciate your position. As it seems your library is the right fit for us, you will most likely see me pop up back here in a few weeks - I hope you won't mind the pestering if your life an work got in the way of happy hacking 😄

Thank you for your help so far.

[FelixSchwarz]

Looking forward to your questions/contributions :-)
Btw: An open source(?) TR069 project sounds interesting. Feel free to include pointers once there is something to see.

[iurisilvio]

It is related to my changes to support relative paths and these imports.

  <xs:import namespace="http://schemas.xmlsoap.org/soap/envelope/" schemaLocation="http://schemas.xmlsoap.org/soap/envelope/"/>
  <xs:import namespace="http://schemas.xmlsoap.org/soap/encoding/" schemaLocation="http://schemas.xmlsoap.org/soap/encoding/"/>

When I had this problem, I just removed these imports and it worked. Of course it is not the fix, just a quick hack.

I'll try to fix it soon.

[ngnpope]

@iurisilvio I have further changes I was working on related to the relative path issues so I'll look at your changes and compare to mine. I should be able to push further updates later today.

[FelixSchwarz]

Based on what @pope1ni said is it ok if I wait a bit with merging?

@iurisilvio: thank you very much for your quick fix, much appreciated.

[iurisilvio]

Yes, wait for @pope1ni. Feel free to use my changes or ignore it if you already did something to fix this issue. 👍

[iurisilvio]

I tested this XSD file today and it failed for another reason. This issue is fixed.

xsd2py https://www.broadband-forum.org/cwmp/cwmp-1-4.xsd

Now it fails with another error.

$ xsd2py https://www.broadband-forum.org/cwmp/cwmp-1-4.xsd
Traceback (most recent call last):
  File "/home/default/.virtualenvs/soapfish/bin/xsd2py", line 9, in <module>
    load_entry_point('soapfish==0.6.0.dev0', 'console_scripts', 'xsd2py')()
  File "/code/soapfish/soapfish/xsd2py.py", line 184, in main
    code = generate_code_from_xsd(xml, encoding='utf-8', cwd=cwd)
  File "/code/soapfish/soapfish/xsd2py.py", line 77, in generate_code_from_xsd
    standalone=standalone)
  File "/code/soapfish/soapfish/xsd2py.py", line 162, in schema_to_py
    return tpl.render(schema=schema, cwd=cwd, base_path=base_path)
  File "/home/default/.virtualenvs/soapfish/local/lib/python2.7/site-packages/Jinja2-2.8-py2.7.egg/jinja2/environment.py", line 989, in render
    return self.environment.handle_exception(exc_info, True)
  File "/home/default/.virtualenvs/soapfish/local/lib/python2.7/site-packages/Jinja2-2.8-py2.7.egg/jinja2/environment.py", line 754, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/code/soapfish/soapfish/templates/xsd", line 11, in top-level template code
    {{- resolve_import(i, known_paths, known_types, schema.targetNamespace, cwd, base_path) }}
  File "/code/soapfish/soapfish/xsd2py.py", line 46, in resolve_import
    xml = open_document(path)
  File "/code/soapfish/soapfish/utils.py", line 38, in open_document
    with open(path, 'rb') as f:
IOError: [Errno 2] No such file or directory: '/code/soapfish/https:/www.broadband-forum.org/cwmp/../../../../../../../../cwmp/cwmp-1-3.xsd'

First the command failed because the site does not accept requests User-Agent. I hard coded changed it just to test the issue.

[ngnpope]

If I recall, the issue we had was to do with whether imported files were local or remote and this would be affected in different ways depending on whether the initial path provided on the command line was local or remote.

In this instance, although the initial path was a remote path, the code originally checked whether the path started with a slash to determine whether it was local. In the remote case, however, starting with a slash indicates that the path is relative to the root of the domain. Worse, if the file is then downloaded from the initial path and used locally, these paths will be incorrect as they should be rewritten to remote paths rather than looking like absolute local paths. But there is probably not much we can do about that...

[iurisilvio]

You are correct. I fixed the problem with the relative paths for remote files, that were handled as local paths.

If you download the file and run it, I think all WSDL/XSD readers will fail too, because you don't have the original URL reference. Maybe we can provide a way to reference it from command line, but I don't think it is a real issue, you have to download all your xsd schema tree.

[ngnpope]

PR #93 was irrelevant to this issue, but PR #96 resolved the issue in the latest stacktrace above.

In addition, I fixed two additional problems in 2ef18c8 and 3ac2a6b that affected the schema file in question.

Now we are blowing up because we always expect <xs:restriction> as a child of <xs:simpleType> but don't handle <xs:union> as a child which contains <xs:restriction> with <xs:enumeration> underneath. Development of support for <xs:union> is being tracked in #38.

I've attached all of the schema files for future reference:

cwmp-1-1.xsd.txt
cwmp-1-2.xsd.txt
cwmp-1-3.xsd.txt
cwmp-1-4.xsd.txt

There are also other weird things that haven't event been considered in some of these such as <xs:redefine>.