JSON Parser #38

bewing · 2017-04-03T22:47:08Z

Basic JSON parser, first profile using it, and a bonus profile template for openconfig-platform.

This branch is based on #28

* develop: Fixed mappers to use real yang names Fixes \napalm-automation#36 Don't assume device_config is a list of strings. Fix indent level in _exec_commands Added ansible examples to the tutorial Added ansible examples Bump version Updating requirements Load dict now update even on defaults and added compliance_report to Root class Conflicts: napalm_yang/parser.py

This parser can understand Python dictionaries from devices and JSON strings injected into native.

bewing · 2017-04-03T22:48:40Z

napalm_yang/mappings/eos/parsers/state/openconfig-platform/components.yaml

+    component:
+        _parse:
+            mode: dict
+            from: "{%- set data = {} -%}{%- for k, v in bookmarks.components.0.0.items() -%}{%- if v is mapping and k is not equalto 'systemInformation' -%}{%- for kk, vv in v.items() -%}{%- do data.update({'{}_{}'.format(k,kk): vv}) -%} {%- endfor -%} {%- endif -%} {%- endfor -%} {{ data|json }}"


@dbarrosop commented in slack that this might be better implemented as a custom filter, which I had not thought of. Should we standardize device output transformation through filters?

If it can be generalized, yes.

_parse_leaf_transform(cls, mapping):

Have a mapping["method"] to call? How would we make sure the method is imported? Looking at how complex transforming the json into the model is, doing the brunt of the transformation in python might be better.

Maybe we should have helpers in the driver classes to transform data for us?

dbarrosop · 2017-04-04T07:22:02Z

I will pull your branch during the weekend and try to use it for parsing interfaces' state before merging.

* develop: Now you can have a closing steatement for list elements Unify naming Removed root BGP and added network instances with all protocols instead Update README.md

* init_native: Now you can have a closing steatement for list elements Unify naming Removed root BGP and added network instances with all protocols instead Update README.md

dbarrosop · 2017-06-03T11:59:03Z

Sorry for the late reply on this. This looks good but CI is broken :(

bewing · 2017-06-03T12:31:20Z

Will look this weekend

Yes, the JSON in matches the JSON out. Go figure.

dbarrosop · 2017-06-06T08:52:26Z

I am holding onto this one for a few more days. I am going to implement some changes that might break your PR but don't worry about that. I will fix them myself and then merge :)

dbarrosop · 2017-06-10T11:17:42Z

Can we decouple the JSONParser from the actual parsing of openconfig-platform? There are some things to discuss there so I wouldn't want to tie them. I'd also appreciate if you could parse the interfaces on EOS for example with this so we can add from the very begining some tests and documentation. If you are constrained with time let me know and we can probably merge this into a separate branch and collaborate.

bewing · 2017-06-11T20:15:39Z

Sure, will rebase today, and work on creating openconfig-interfaces eos profile

dbarrosop · 2017-06-12T09:40:33Z

Great, thanks. Don't try to parse too much, something similar to the state parsing for junos interfaces is more than enough. I just want to have something to test on, specially as I might be refactoring some bits in the upcoming weeks. I am trying to make things a bit more efficient and simple to use.

* develop: (74 commits) add post-process-filter attribute to text parser (napalm-automation#59) Allow each parser to modify native on start (napalm-automation#28) Updated CHANGELOG Updated docs Removed negate_recursively as it's unnecessary Added network-instance/junos translator Added reuse and replace_on_merge When using lists of actions allow advancing containers in xml Added _translate_leaf_map Added config parser for network-instances/junos Added missing key Various improvement to xml parser Minor fix to network-instance/eos Added tests for network-instance/eos Added network-instance parser/translator for EOS We can now remove elements recursively in cases where the nesting of YANG model doesnt reflect the nesting in native configuration Minor fixes to the logic of the lists Propagate attribute name rather than yang_name Adapted network instances model to lists Find translation point per rule as before ... Conflicts: napalm_yang/helpers.py napalm_yang/parser.py

bewing · 2017-06-17T17:22:02Z

Jesus that merge took longer to troubleshoot than I thought it would. Will work on replacing openconfig-platform with openconfig-interfaces for test case purposes next.

bewing · 2017-06-18T00:07:24Z

Couple of notes:

Couldn't figure out how to cross-reference in the parser -- IE, couldn't use parent_key from interfaces.interface as a dictionary key in bookmarks.interfaces.1, to collect ifIndex
Looks like the test interface doesn't really support multiple datasources? Might want to have a discussion regarding that.

bewing · 2017-06-18T00:08:24Z

Further note -- this profile is mostly just for testing, as it cannot correctly handle subinterfaces, which show up as regular interfaces in native. :/

dbarrosop · 2017-06-18T15:00:56Z

It's alright. I am going to merge it as I want to re-do some parts of the whole parser framework and I will break stuff so having it merged will allow me to fix it myself rather than having to ask you again to do it yourself.

In case you are curious, I want to do three main changes to the framework:

from will work like in in the translator. Which means that it will be only necessary to specify when you don't want to follow the "expected workflow" (if you capture an interface block the expectation will be that everything underneath will be working with that captured block so you will use from only if you want to use some other block instead)
The bookmarks will work with references to objects rather than with strings, similar to how it works in the translator. I hope this will avoid a lot of serialization/deserialization and improve performance.
mode will only be required when you don't want to use the "default" mode.

bewing added 10 commits March 29, 2017 14:46

Allow each parser to modify native on start

5f47134

Use builtins super()

d074a5d

Move parser.init_native(), revert some assumptions.

d75bc7c

Don't forget pylama

73a66ee

Update bookmark when running init_native

e8a40e1

Syntax

75cfc84

Add JSONParser to handle JSON objects

9863d42

This parser can understand Python dictionaries from devices and JSON strings injected into native.

Add components state template

154253d

EOS oc-platform framework and test

073a949

bewing commented Apr 3, 2017

View reviewed changes

bewing added 2 commits April 4, 2017 10:10

Fix leaf parser type

9d54b26

More platform parsing

0f1fde9

bewing mentioned this pull request Apr 5, 2017

Discussion: Complex models (oc-platform) #39

Closed

bewing added 3 commits April 11, 2017 10:10

Merge branch 'develop' into init_native

4580e30

* develop: Now you can have a closing steatement for list elements Unify naming Removed root BGP and added network instances with all protocols instead Update README.md

Merge branch 'init_native' into JSONParser

0f19771

* init_native: Now you can have a closing steatement for list elements Unify naming Removed root BGP and added network instances with all protocols instead Update README.md

Update profile for unified naming

3bdf501

dbarrosop self-assigned this Apr 16, 2017

Update ex. profile for native dict

e09a3dd

Update unit tests

2095966

Yes, the JSON in matches the JSON out. Go figure.

mirceaulinic approved these changes Jun 4, 2017

View reviewed changes

dbarrosop mentioned this pull request Jun 6, 2017

Allow pure python parsers for complex models #54

Closed

bewing mentioned this pull request Jun 6, 2017

WIP: Add _oc_platform private method napalm-automation/napalm-eos#165

Closed

bewing added 2 commits June 17, 2017 18:31

Change value mapping

6f448f7

Test case for JSON Parser and EOS interfaces

3478d58

dbarrosop merged commit 80cd38f into napalm-automation:develop Jun 18, 2017

bewing deleted the JSONParser branch June 18, 2017 15:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JSON Parser #38

JSON Parser #38

bewing commented Apr 3, 2017

bewing Apr 3, 2017

dbarrosop Apr 4, 2017

bewing Apr 4, 2017 •

edited

Loading

bewing Apr 4, 2017

dbarrosop commented Apr 4, 2017

dbarrosop commented Jun 3, 2017

bewing commented Jun 3, 2017 via email •

edited

Loading

dbarrosop commented Jun 6, 2017

dbarrosop commented Jun 10, 2017

bewing commented Jun 11, 2017

dbarrosop commented Jun 12, 2017

bewing commented Jun 17, 2017

bewing commented Jun 18, 2017

bewing commented Jun 18, 2017

dbarrosop commented Jun 18, 2017

JSON Parser #38

JSON Parser #38

Conversation

bewing commented Apr 3, 2017

bewing Apr 3, 2017

Choose a reason for hiding this comment

dbarrosop Apr 4, 2017

Choose a reason for hiding this comment

bewing Apr 4, 2017 • edited Loading

Choose a reason for hiding this comment

bewing Apr 4, 2017

Choose a reason for hiding this comment

dbarrosop commented Apr 4, 2017

dbarrosop commented Jun 3, 2017

bewing commented Jun 3, 2017 via email • edited Loading

dbarrosop commented Jun 6, 2017

dbarrosop commented Jun 10, 2017

bewing commented Jun 11, 2017

dbarrosop commented Jun 12, 2017

bewing commented Jun 17, 2017

bewing commented Jun 18, 2017

bewing commented Jun 18, 2017

dbarrosop commented Jun 18, 2017

bewing Apr 4, 2017 •

edited

Loading

bewing commented Jun 3, 2017 via email •

edited

Loading