Module "ietf-yang-library" parsing failed

[fabioantonini1969]

Hi
I have downloaded libyang 0.10.210 and now I have problem to parse ietf-yang-library. For instance yanglint fails to start with the following traces.
Of course any other application using libyang fails to start with the same problem

~ # yanglint
libyang[2]: Reading module "yang".
libyang[2]: Module "yang" successfully parsed.
libyang[2]: Reading module "ietf-inet-types".
libyang[2]: Resolving derived type "union" failed, it will be attempted later.
libyang[2]: Resolving type default "(null)" failed, it will be attempted later.
libyang[2]: Resolving derived type "union" failed, it will be attempted later.
libyang[2]: Resolving type default "(null)" failed, it will be attempted later.
libyang[2]: Resolving derived type "union" failed, it will be attempted later.
libyang[2]: Resolving type default "(null)" failed, it will be attempted later.
libyang[2]: Resolving derived type "union" failed, it will be attempted later.
libyang[2]: Resolving type default "(null)" failed, it will be attempted later.
libyang[2]: Resolving "ietf-inet-types" unresolved schema nodes and their constraints...
libyang[2]: All "ietf-inet-types" schema nodes and constraints resolved.
libyang[2]: Module "ietf-inet-types" successfully parsed.
libyang[2]: Reading module "ietf-yang-types".
libyang[2]: Module "ietf-yang-types" successfully parsed.
libyang[2]: Reading module "ietf-yang-library".
libyang[2]: Resolving uses "common-leafs" failed, it will be attempted later.
libyang[2]: Resolving uses "schema-leaf" failed, it will be attempted later.
libyang[2]: Resolving uses "common-leafs" failed, it will be attempted later.
libyang[2]: Resolving list keys "name revision" failed, it will be attempted later.
libyang[2]: Resolving uses "common-leafs" failed, it will be attempted later.
libyang[2]: Resolving uses "schema-leaf" failed, it will be attempted later.
libyang[2]: Resolving list keys "name revision" failed, it will be attempted later.
libyang[2]: Resolving list keys "name revision" failed, it will be attempted later.
libyang[2]: Resolving uses "module-list" failed, it will be attempted later.
libyang[2]: Resolving "ietf-yang-library" unresolved schema nodes and their constraints...
libyang[0]: Module "ietf-yang-library" parsing failed.
Failed to create context.

What's happening?
The problem occurs also with 0.10.211 downloaded this morning.
It doesn't occur with the 0.9.186
What am I missing?
Thanks a lot
fabio

[fabioantonini1969]

it can be tagged as "Question"

[fabioantonini1969]

Just for your information I'm cross-compiling libyang for a MIPS target. Not a x86 platform.

[fabioantonini1969]

I have some more verbose traces (ly_verb(LY_LLDBG))

libyang[2]: Reading module "ietf-yang-library".
libyang[3]: DICT: inserting (refcount) "urn:ietf:params:xml:ns:yang:ietf-yang-library"
libyang[3]: DICT: inserting (refcount) "yanglib"
libyang[3]: DICT: inserting (refcount) "IETF NETCONF (Network Configuration) Working Group"
libyang[3]: DICT: inserting (refcount) "WG Web: https://datatracker.ietf.org/wg/netconf/
WG List: mailto:netconf@ietf.org

WG Chair: Mehmet Ersue
mailto:mehmet.ersue@nsn.com

WG Chair: Mahesh Jethanandani
mailto:mjethanandani@gmail.com

Editor: Andy Bierman
mailto:andy@yumaworks.com

Editor: Martin Bjorklund
mailto:mbj@tail-f.com

Editor: Kent Watsen
mailto:kwatsen@juniper.net"
libyang[3]: DICT: inserting (refcount) "This module contains monitoring information about the YANG
modules and submodules that are used within a YANG-based
server.

Copyright (c) 2016 IETF Trust and the persons identified as
authors of the code. All rights reserved.

Redistribution and use in source and binary forms, with or
without modification, is permitted pursuant to, and subject
to the license terms contained in, the Simplified BSD License
set forth in Section 4.c of the IETF Trust's Legal Provisions
Relating to IETF Documents
(http://trustee.ietf.org/license-info).

This version of this YANG module is part of RFC 7895; see
the RFC itself for full legal notices."
libyang[3]: DICT: inserting (refcount) "yang"
libyang[3]: DICT: inserting (refcount) "inet"
libyang[3]: DICT: inserting (refcount) "Initial revision."
libyang[3]: DICT: inserting (refcount) "RFC 7895: YANG Module Library."
libyang[3]: DICT: inserting (refcount) "revision-identifier"
libyang[3]: DICT: inserting (refcount) "Represents a specific date in YYYY-MM-DD format."
libyang[3]: DICT: inserting (refcount) "string"
libyang[3]: DICT: inserting "�\d{4}-\d{2}-\d{2}"
libyang[3]: DICT: inserting (refcount) "module-list"
libyang[3]: DICT: inserting (refcount) "The module data structure is represented as a grouping
so it can be reused in configuration or another monitoring
data structure."
libyang[3]: YIN: parsing grouping statement "module-list"
libyang[3]: DICT: inserting (refcount) "common-leafs"
libyang[3]: DICT: inserting (refcount) "Common parameters for YANG modules and submodules."
libyang[3]: YIN: parsing grouping statement "common-leafs"
libyang[3]: DICT: inserting (refcount) "name"
libyang[3]: DICT: inserting (refcount) "The YANG module or submodule name."
libyang[3]: YIN: parsing leaf statement "name"
libyang[3]: DICT: inserting "ietf-yang-types:yang-identifier"
libyang[3]: DICT: inserting (refcount) "ietf-yang-types"
libyang[3]: DICT: inserting (refcount) "revision"
libyang[3]: DICT: inserting (refcount) "The YANG module or submodule revision date.
A zero-length string is used if no revision statement
is present in the YANG module or submodule."
libyang[3]: YIN: parsing leaf statement "revision"
libyang[3]: DICT: inserting (refcount) "union"
libyang[3]: DICT: inserting (refcount) "revision-identifier"
libyang[3]: DICT: inserting (refcount) "string"
libyang[3]: DICT: inserting (refcount) "0"
libyang[3]: DICT: inserting (refcount) "schema-leaf"
libyang[3]: DICT: inserting (refcount) "Common schema leaf parameter for modules and submodules."
libyang[3]: YIN: parsing grouping statement "schema-leaf"
libyang[3]: DICT: inserting (refcount) "schema"
libyang[3]: DICT: inserting (refcount) "Contains a URL that represents the YANG schema
resource for this module or submodule.

This leaf will only be present if there is a URL
available for retrieval of the schema for this entry."
libyang[3]: YIN: parsing leaf statement "schema"
libyang[3]: DICT: inserting "ietf-inet-types:uri"
libyang[3]: DICT: inserting (refcount) "ietf-inet-types"
libyang[3]: DICT: inserting (refcount) "module"
libyang[3]: DICT: inserting (refcount) "Each entry represents one revision of one module
currently supported by the server."
libyang[3]: YIN: parsing list statement "module"
libyang[3]: DICT: inserting (refcount) "common-leafs"
libyang[3]: YIN: parsing uses statement "common-leafs"
libyang[2]: Resolving uses "common-leafs" failed, it will be attempted later.
libyang[3]: DICT: inserting (refcount) "schema-leaf"
libyang[3]: YIN: parsing uses statement "schema-leaf"
libyang[2]: Resolving uses "schema-leaf" failed, it will be attempted later.
libyang[3]: DICT: inserting (refcount) "namespace"
libyang[3]: DICT: inserting (refcount) "The XML namespace identifier for this module."
libyang[3]: YIN: parsing leaf statement "namespace"
libyang[3]: DICT: inserting "ietf-inet-types:uri"
libyang[3]: DICT: inserting (refcount) "ietf-inet-types"
libyang[3]: DICT: inserting (refcount) "feature"
libyang[3]: DICT: inserting (refcount) "List of YANG feature names from this module that are
supported by the server, regardless of whether they are
defined in the module or any included submodule."
libyang[3]: YIN: parsing leaf-list statement "feature"
libyang[3]: DICT: inserting "ietf-yang-types:yang-identifier"
libyang[3]: DICT: inserting (refcount) "ietf-yang-types"
libyang[3]: DICT: inserting (refcount) "deviation"
libyang[3]: DICT: inserting (refcount) "List of YANG deviation module names and revisions
used by this server to modify the conformance of
the module associated with this entry. Note that
the same module can be used for deviations for
multiple modules, so the same entry MAY appear
within multiple 'module' entries.

The deviation module MUST be present in the 'module'
list, with the same name and revision values.
The 'conformance-type' value will be 'implement' for
the deviation module."
libyang[3]: YIN: parsing list statement "deviation"
libyang[3]: DICT: inserting (refcount) "common-leafs"
libyang[3]: YIN: parsing uses statement "common-leafs"
libyang[2]: Resolving uses "common-leafs" failed, it will be attempted later.
libyang[3]: DICT: inserting (refcount) "name revision"
libyang[2]: Resolving list keys "name revision" failed, it will be attempted later.
libyang[3]: DICT: inserting (refcount) "conformance-type"
libyang[3]: DICT: inserting (refcount) "Indicates the type of conformance the server is claiming
for the YANG module identified by this entry."
libyang[3]: YIN: parsing leaf statement "conformance-type"
libyang[3]: DICT: inserting (refcount) "enumeration"
libyang[3]: DICT: inserting (refcount) "implement"
libyang[3]: DICT: inserting (refcount) "Indicates that the server implements one or more
protocol-accessible objects defined in the YANG module
identified in this entry. This includes deviation
statements defined in the module.

For YANG version 1.1 modules, there is at most one
module entry with conformance type 'implement' for a
particular module name, since YANG 1.1 requires that,
at most, one revision of a module is implemented.

For YANG version 1 modules, there SHOULD NOT be more
than one module entry for a particular module name."
libyang[3]: DICT: inserting (refcount) "import"
libyang[3]: DICT: inserting (refcount) "Indicates that the server imports reusable definitions
from the specified revision of the module but does
not implement any protocol-accessible objects from
this revision.

Multiple module entries for the same module name MAY
exist. This can occur if multiple modules import the
same module but specify different revision dates in
the import statements."
libyang[3]: DICT: inserting (refcount) "submodule"
libyang[3]: DICT: inserting (refcount) "Each entry represents one submodule within the
parent module."
libyang[3]: YIN: parsing list statement "submodule"
libyang[3]: DICT: inserting (refcount) "common-leafs"
libyang[3]: YIN: parsing uses statement "common-leafs"
libyang[2]: Resolving uses "common-leafs" failed, it will be attempted later.
libyang[3]: DICT: inserting (refcount) "schema-leaf"
libyang[3]: YIN: parsing uses statement "schema-leaf"
libyang[2]: Resolving uses "schema-leaf" failed, it will be attempted later.
libyang[3]: DICT: inserting (refcount) "name revision"
libyang[2]: Resolving list keys "name revision" failed, it will be attempted later.
libyang[3]: DICT: inserting (refcount) "name revision"
libyang[2]: Resolving list keys "name revision" failed, it will be attempted later.
libyang[3]: DICT: inserting (refcount) "modules-state"
libyang[3]: DICT: inserting (refcount) "Contains YANG module monitoring information."
libyang[3]: YIN: parsing container statement "modules-state"
libyang[3]: DICT: inserting (refcount) "module-set-id"
libyang[3]: DICT: inserting (refcount) "Contains a server-specific identifier representing
the current set of modules and submodules. The
server MUST change the value of this leaf if the
information represented by the 'module' list instances
has changed."
libyang[3]: YIN: parsing leaf statement "module-set-id"
libyang[3]: DICT: inserting (refcount) "string"
libyang[3]: DICT: inserting (refcount) "module-list"
libyang[3]: YIN: parsing uses statement "module-list"
libyang[2]: Resolving uses "module-list" failed, it will be attempted later.
libyang[3]: DICT: inserting (refcount) "yang-library-change"
libyang[3]: DICT: inserting (refcount) "Generated when the set of modules and submodules supported
by the server has changed."
libyang[3]: YIN: parsing notification statement "yang-library-change"
libyang[3]: DICT: inserting (refcount) "module-set-id"
libyang[3]: DICT: inserting (refcount) "Contains the module-set-id value representing the
set of modules and submodules supported at the server at
the time the notification is generated."
libyang[3]: YIN: parsing leaf statement "module-set-id"
libyang[3]: DICT: inserting (refcount) "leafref"
libyang[3]: DICT: inserting "/ietf-yang-library:modules-state/ietf-yang-library:module-set-id" with collision
libyang[2]: Resolving "ietf-yang-library" unresolved schema nodes and their constraints...
libyang[0]: Module "ietf-yang-library" parsing failed.
Failed to create context.

hope this can help

[fabioantonini1969]

Now I have tried the version 0.9.186 and it works

/usr/lib # yanglint
libyang[3]: Reading module "yang".
libyang[3]: Module "yang" successfully parsed.
libyang[3]: Reading module "ietf-inet-types".
libyang[3]: Resolving derived type "union" failed, it will be attempted later.
libyang[3]: Resolving derived type "union" failed, it will be attempted later.
libyang[3]: Resolving derived type "union" failed, it will be attempted later.
libyang[3]: Resolving unresolved schema nodes and their constraints...
libyang[3]: All schema nodes and constraints resolved.
libyang[3]: Module "ietf-inet-types" successfully parsed.
libyang[3]: Reading module "ietf-yang-types".
libyang[3]: Module "ietf-yang-types" successfully parsed.
libyang[3]: Reading module "ietf-yang-library".
libyang[3]: Module "ietf-yang-library" successfully parsed.

[michalvasko]

Hi,
it's really difficult for us to help you if you're not using x86. Unfortunately, there is no relevant information in the debug output you provided, which is strange in itself. Are you able to do some debugging yourself (using gdb for example)?

Regards,
Michal

[fabioantonini]

Hi Michal
Thanks a lot for your feedback
It's strange that with the previous version 0.9.186 everything worked as a charm.
I don't apply any custom patch. Just compile with my cross-compiler for MIPS. So I cannot believe that this is an endianess problem. I'll try to do by myself. thanks
fabio

[fabioantonini]

just a last comment about some output strings
is it normal to have such a kind of output?

libyang[3]: DICT: inserting "�(([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5]).){3}([0-9]|[1-9][0-9]|1[0-9][0-9]|2[0-4][0-9]|25[0-5])(%[\p{N}\p{L}]+)?" with collision

thanks
fabio

[michalvasko]

Hi Fabio,
you are right, the first character is obviously some invalid character value, it should not be there, so you can start you debugging there.

Regards,
Michal

[fabioantonini]

does this new version depend on a specific release of pcre?
I cannot understand what's the problem. The compiler is always the same. One week ago the libyang.so worked fine. I have just updated the libyang release. So something has changed in the source code.
I don't modify anything in the source code for MIPS porting.
regards
fabio

[michalvasko]

Hi Fabio,
no, required PCRE version remained the same, but there are lots of changes, we support YANG 1.1 now. But something is obviusly wrong in the patterns being added into the dictionary, try to solve that problem first.

Regards,
Michal

[fabioantonini1969]

Hi Michal
just to let me to be aligned with you, which is the libpcre version are you using?
thanks
fabio

[rkrejci]

I don't think that libpcre is problem, I would focus on that odd byte in this string:

libyang[3]: DICT: inserting "�\d{4}-\d{2}-\d{2}"

I plan to look at that on 32b system, on 64b everything looks fine, neither the source file contains that byte, so it must be something in parsing XML. There are some changes in 0.10 connected with more strict rules in YANG 1.1, but it should be just about checking, not about parsing. I will see later today.

[fabioantonini]

Hi Radek
thanks a lot for your support
my processor is a 32b system (embedded mips cpu). Feel free to ask more details about my platform if you need. At the moment I had to go back to the previous running version (0.9.186) and also for sysrepo/netopeer2. Anyway I should move to the newest release of all packages as soon as this problem will be focused.
regards
fabio

[rkrejci]

Hi, I'm not able to reproduce it on 32b i386. Could you try to compile and run it in your environment with the modified libyang's parser.c and yanglint's main.c? It adds a lot of debug messages. Please run the compiled yanglint:

$ ./yanglint 2> log.txt

and provide the log (github allows you to attach file to the comment).

modifiedsources.zip

[fabioantonini]

Hi Radek
but what debug level is
ly_verb(5)
in libyang.h I read
typedef enum {
LY_LLSILENT = -1, /< Print no messages. */
LY_LLERR = 0, /< Print only error messages, default value. /
LY_LLWRN, /< Print error and warning messages. */
LY_LLVRB, /< Besides errors and warnings, print some other verbose messages. */
LY_LLDBG /*< Print all messages including some development debug messages. */
} LY_LOG_LEVEL;

Anyway running the modified yanglint with the modified libyang.so.0.10.216 I get the following output

~ # cat log
libyang[2]: Reading module "yang".
libyang[2]: Module "yang" successfully parsed.
libyang[2]: Reading module "ietf-inet-types".
libyang[2]: Resolving derived type "union" failed, it will be attempted later.
libyang[2]: Resolving type default "(null)" failed, it will be attempted later.
libyang[2]: Resolving derived type "union" failed, it will be attempted later.
libyang[2]: Resolving type default "(null)" failed, it will be attempted later.
libyang[2]: Resolving derived type "union" failed, it will be attempted later.
libyang[2]: Resolving type default "(null)" failed, it will be attempted later.
libyang[2]: Resolving derived type "union" failed, it will be attempted later.
libyang[2]: Resolving type default "(null)" failed, it will be attempted later.
libyang[2]: Resolving "ietf-inet-types" unresolved schema nodes and their constraints...
libyang[2]: All "ietf-inet-types" schema nodes and constraints resolved.
libyang[2]: Module "ietf-inet-types" successfully parsed.
libyang[2]: Reading module "ietf-yang-types".
libyang[2]: Module "ietf-yang-types" successfully parsed.
libyang[2]: Reading module "ietf-yang-library".
libyang[2]: Resolving uses "common-leafs" failed, it will be attempted later.
libyang[2]: Resolving uses "schema-leaf" failed, it will be attempted later.
libyang[2]: Resolving uses "common-leafs" failed, it will be attempted later.
libyang[2]: Resolving list keys "name revision" failed, it will be attempted later.
libyang[2]: Resolving uses "common-leafs" failed, it will be attempted later.
libyang[2]: Resolving uses "schema-leaf" failed, it will be attempted later.
libyang[2]: Resolving list keys "name revision" failed, it will be attempted later.
libyang[2]: Resolving list keys "name revision" failed, it will be attempted later.
libyang[2]: Resolving uses "module-list" failed, it will be attempted later.
libyang[2]: Resolving "ietf-yang-library" unresolved schema nodes and their constraints...
libyang[0]: Module "ietf-yang-library" parsing failed.
Failed to create context.
regards
fabio

p.s. my cpu architecture is big endian. Could be this the problem?