Convert to Python 3

[rw-access]

Getting ahead of python2's EOL. There were a few minor changes, and I believe that whitespace changes in json.dump, and dictionary default ordering may have changed some of the autogenerated code.

Unit tests are still passing, although there's one deprecation warning that's easy to address

build/ve/bin/python -m unittest discover --start-directory scripts/tests
..
ecs/scripts/tests/test_ecs_helpers.py:72: DeprecationWarning: Please use assertEqual instead.
  'base': {'group': 1, 'name': 'base'}
........................
----------------------------------------------------------------------

If you disable whitespace changes, it'll be easier to view the diff.

[rw-access]

I renamed this to avoid variable shadowing, since dict already means something.
It wasn't strictly necessary, but it makes the code easier to read, and makes PyCharm more pleased with me.

[rw-access]

We got lucky before, because with Python 2, the ordering was deterministic

[rw-access]

The previous version was passing unit tests but generated code that was inconsistent, because "type": "object" was missing. I can make it so this is always present, but I don't exactly know the intendend behavior.

ecs/generated/elasticsearch/7/template.json

Lines 419 to 496 in 94e7376

	"dns": {
	"properties": {
	"answers": {
	"properties": {
	"class": {
	"ignore_above": 1024,
	"type": "keyword"
	},
	"data": {
	"ignore_above": 1024,
	"type": "keyword"
	},
	"name": {
	"ignore_above": 1024,
	"type": "keyword"
	},
	"ttl": {
	"type": "long"
	},
	"type": {
	"ignore_above": 1024,
	"type": "keyword"
	}
	}
	},
	"header_flags": {
	"ignore_above": 1024,
	"type": "keyword"
	},
	"id": {
	"ignore_above": 1024,
	"type": "keyword"
	},
	"op_code": {
	"ignore_above": 1024,
	"type": "keyword"
	},
	"question": {
	"properties": {
	"class": {
	"ignore_above": 1024,
	"type": "keyword"
	},
	"name": {
	"ignore_above": 1024,
	"type": "keyword"
	},
	"registered_domain": {
	"ignore_above": 1024,
	"type": "keyword"
	},
	"subdomain": {
	"ignore_above": 1024,
	"type": "keyword"
	},
	"top_level_domain": {
	"ignore_above": 1024,
	"type": "keyword"
	},
	"type": {
	"ignore_above": 1024,
	"type": "keyword"
	}
	}
	},
	"resolved_ip": {
	"type": "ip"
	},
	"response_code": {
	"ignore_above": 1024,
	"type": "keyword"
	},
	"type": {
	"ignore_above": 1024,
	"type": "keyword"
	}
	}
	},

Replacing these lines

ecs/scripts/generators/es_template.py

Lines 27 to 30 in 94e7376

	if current_nesting not in dict:
	dict[current_nesting] = {'properties': {}}
	elif 'type' in dict[current_nesting] and 'object' == dict[current_nesting]['type']:
	dict[current_nesting] = {'type': dict[current_nesting]['type'], 'properties': {}}

with this should do what I think is our intended behavior, but it's very clear to me what that is.

        dct[current_nesting].setdefault("properties", {})

[webmat]

Yeah iirc it's a nasty bit that I added when adding dns.answers. Adding type: object everywhere we have further nesting would be fine and a noop, as far as I understand.

I'd prefer to keep it as it was, and only handle the explicit type: object in place when it's already there, for now.

[webmat]

Huge thanks for this! I'll review shortly :-)

I'm specifically using OrderedDict in a few strategic places. But the default dict becoming ordered does show up in a few more places, like the labels example :-)

[rud]

Tiny suggestion: update the python version in CONTRIBUTING.md.

[webmat]

The build is now failing because the Golang base image for 1.13.x comes with 5 year old Python 3.5.2. Not sure why the Travis config for python 3.7 isn't kicking in 🤦‍♂

[webmat]

Related: #630

[webmat]

The only oddity left is around the two timestamp field's examples. I had never noticed that it was getting interpreted before. It was unquoted in the YAML source files, and was being changed slightly by Python 2.7. Upgrading to 3.6+ was changing it slightly, but in a different manner.

event.created:

2.7

• source value was: 2016-05-23T08:05:34.857Z
• output value (e.g. in asciidoc) was: 2016-05-23 08:05:35.101000

3.6/3.7

• source value was: 2016-05-23T08:05:34.857Z
• output value (e.g. in asciidoc) was: 2016-05-23 08:05:34.857000+00:00

I've quoted the example in the source YAML, to avoid the timestamp being interpreted. In the end, ECS is not actually mandating a precise timestamp format. So this is different than before, but this makes the timestamp examples consistent with what's in the source YAML everywhere.

Now:

• source value is: '2016-05-23T08:05:34.857Z'
• output value (e.g. in asciidoc) is: 2016-05-23T08:05:34.857Z

[webmat]

@rw-access @ruflin This PR is ready for final review.

[rw-access]

thanks for the updates @webmat. LGTM