Feature/event factory

[kevinschoon]

Heya --

I have added support for Marathon's EventBus as well as adding a few integration tests. Unfortunately I need Python3 support and I can't seem to get the existing setup.py configuration to work for me. I ran 2to3 directly against the repo and then made a few import changes. I tested running with Python2 and Python3 and it seems to work ok.

Thank you for all your work on this library!

Kevin

[mbabineau]

This is truly awesome. Thank you.

I don't have a Marathon server handy at the moment, but my friend @pez5001 ran the tests and got the following:

paul-/tmp/marathon-python(feature/event_factory)$ python -m unittest -v

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK

Any idea why that might be?

[kevinschoon]

Hmmm... pretty weird but it seems outside of my IDE (PyCharm) the unit tests don't load under Python2.. but from my IDE they run fine on both Python2 and Python3.

I can take a closer look later today.

From CLI I just tested with a new virtualenv:

cd marathon-python
virtualenv -p `which python3` ~/.py34
source ~/.py34/bin/activate
python setup.py develop
python -m unittest -v

(.py34)kevin@mbp$ python -m unittest -v
test_create (tests.test_create_app.TestCreateApp) ... ok
test_event_factory (tests.test_events.TestEventSubscription) ... Server listening on port 9991
192.168.99.110 - - [04/Jun/2015 15:58:01] "POST / HTTP/1.1" 200 -
Processed 1 requests
192.168.99.110 - - [04/Jun/2015 15:58:01] "POST / HTTP/1.1" 200 -
Processed 2 requests
MarathonSubscribeEvent::{"callbackUrl": "http://192.168.99.1:9991", "clientIp": "192.168.99.1", "eventType": "subscribe_event", "timestamp": "2015-06-04T19:58:01.585Z"} ['KNOWN_ATTRIBUTES', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_set', 'attribute_name_to_marathon_object', 'callback_url', 'client_ip', 'event_type', 'from_json', 'json_repr', 'timestamp', 'to_json'] {'timestamp': '2015-06-04T19:58:01.585Z', 'callback_url': 'http://192.168.99.1:9991', 'event_type': 'subscribe_event', 'client_ip': '192.168.99.1'}
ok

----------------------------------------------------------------------
Ran 2 tests in 82.347s

OK

From my IDE with Python 2.7:

/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/bin/python2.7 "/Applications/PyCharm CE.app/Contents/helpers/pycharm/utrunner.py" /Users/kevin/repos/greencase/marathon-python/ true
Testing started at 3:49 PM ...
Server listening on port 9991
Processed 1 requests
Processed 2 requests
192.168.99.110 - - [04/Jun/2015 15:50:40] "POST / HTTP/1.1" 200 -
192.168.99.110 - - [04/Jun/2015 15:50:40] "POST / HTTP/1.1" 200 -
Processed 3 requests
192.168.99.110 - - [04/Jun/2015 15:50:40] "POST / HTTP/1.1" 200 -
(MarathonSubscribeEvent::{"callbackUrl": "http://192.168.99.1:9991", "clientIp": "192.168.99.1", "eventType": "subscribe_event", "timestamp": "2015-06-04T19:50:40.014Z"}, ['KNOWN_ATTRIBUTES', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_set', 'attribute_name_to_marathon_object', 'callback_url', 'client_ip', 'event_type', 'from_json', 'json_repr', 'timestamp', 'to_json'], {'timestamp': u'2015-06-04T19:50:40.014Z', 'client_ip': u'192.168.99.1', 'callback_url': u'http://192.168.99.1:9991', 'event_type': u'subscribe_event'})

[kevinschoon]

Just took a look at this and you just need to add the "discover" argument. @pez5001 Let me know if this works for you.

Thanks!

cd marathon-python
python -m unittest discover

.Server listening on port 9999
192.168.99.110 - - [08/Jun/2015 10:26:20] "POST / HTTP/1.1" 200 -
Processed 1 requests
192.168.99.110 - - [08/Jun/2015 10:26:20] "POST /events HTTP/1.1" 200 -
Processed 2 requests
(MarathonSubscribeEvent::{"callbackUrl": "http://192.168.99.1:9999", "clientIp": "192.168.99.1", "eventType": "subscribe_event", "timestamp": "2015-06-08T14:26:20.671Z"}, ['KNOWN_ATTRIBUTES', '__class__', '__delattr__', '__dict__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_set', 'attribute_name_to_marathon_object', 'callback_url', 'client_ip', 'event_type', 'from_json', 'json_repr', 'timestamp', 'to_json'], {'timestamp': u'2015-06-08T14:26:20.671Z', 'client_ip': u'192.168.99.1', 'callback_url': u'http://192.168.99.1:9999', 'event_type': u'subscribe_event'})
.
----------------------------------------------------------------------
Ran 2 tests in 21.915s

OK

[pez5001]

It looks like these tests only work if the test machine and marathon machine are in the same network... a callback URL of http://192.168.99.1:9999 isn't going to work on an ec2-hosted mesos+marathon cluster if I run the tests from my laptop inside my home or work network, right?

[kevinschoon]

No you would need to expose the callback interface to the internet, perhaps you could forward via SSH?

[pez5001]

I ran the tests... it launched a test app in mesos, then died with this error:

======================================================================
ERROR: test_event_factory (tests.test_events.TestEventSubscription)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/marathon-python-feature-event_factory/tests/test_events.py", line 36, in test_event_factory
    o = self.factory.process(queue.get(timeout=120))  # Raise queueEmpty if the test exceeds 2 minutes.
  File "/home/ubuntu/marathon-python-feature-event_factory/marathon/models/events.py", line 121, in process
    return clazz.from_json(event)
  File "/home/ubuntu/marathon-python-feature-event_factory/marathon/models/base.py", line 30, in from_json
    return cls(**{to_snake_case(k): v for k,v in attributes.items()})
  File "/home/ubuntu/marathon-python-feature-event_factory/marathon/models/events.py", line 30, in __init__
    self._set(attribute, kwargs[attribute])
  File "/home/ubuntu/marathon-python-feature-event_factory/marathon/models/events.py", line 37, in _set
    attribute = clazz.from_json(attribute)  # If this attribute already has a Marathon object instantiate it.
  File "/home/ubuntu/marathon-python-feature-event_factory/marathon/models/base.py", line 30, in from_json
    return cls(**{to_snake_case(k): v for k,v in attributes.items()})
TypeError: __init__() got an unexpected keyword argument 'ignore_http1xx'

----------------------------------------------------------------------

[kevinschoon]

Maybe you're running a different version of Marathon?

Mine is 0.8.1

[pez5001]

That could be it... I'm on 0.8.2-RC3

[mrtheb]

For reference, the error mentioned above about ignore_http1xx should be fixed with de5c91a. Can you try with latest?

[pez5001]

That fixed it for me. Tests passing now 👍

[mbabineau]

👍 👍 👍