Skip to content

Support data gzip compression in EmonHubEmoncmsHTTPInterfacer #70

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

Support data gzip compression in EmonHubEmoncmsHTTPInterfacer #70

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 25 additions & 4 deletions src/interfacers/EmonHubEmoncmsHTTPInterfacer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@
"""
import time
import json
import urllib
import zlib
from emonhub_interfacer import EmonHubInterfacer

class EmonHubEmoncmsHTTPInterfacer(EmonHubInterfacer):
Expand All @@ -14,7 +16,7 @@ def __init__(self, name):
# defaults previously defined in inherited emonhub_interfacer
# here we are just changing the batchsize from 1 to 100
# and the interval from 0 to 30
self._defaults.update({'batchsize': 100,'interval': 30})
self._defaults.update({'batchsize': 100, 'interval': 30})
# This line will stop the default values printing to logfile at start-up
self._settings.update(self._defaults)

Expand All @@ -23,7 +25,8 @@ def __init__(self, name):
'apikey': "",
'url': "http://emoncms.org",
'senddata': 1,
'sendstatus': 0
'sendstatus': 0,
'compress': 1
}

# set an absolute upper limit for number of items to process per post
Expand All @@ -44,7 +47,6 @@ def _process_post(self, databuffer):
# Return true to clear buffer if the apikey is not set
return True


data_string = json.dumps(databuffer, separators=(',', ':'))

# Prepare URL string of the form
Expand All @@ -57,7 +59,17 @@ def _process_post(self, databuffer):

# Construct post_url (without apikey)
post_url = self._settings['url']+'/input/bulk'+'.json?apikey='
post_body = "data="+data_string+"&sentat="+str(sentat)

# Construct post body
post_body_data = {"data": data_string, "sentat": sentat}

if self._settings['compress']:
# Compress data and encode as hex string.
post_body_data["data"] = zlib.compress(post_body_data["data"]).encode("hex")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is throwing away the sentat key. I don't know if that's important.

More importantly this needs a bit of tweaking to make it work in Python3. The argument to zlib.compress must be bytes, so we need to encode it. compress then returns more bytes, but you need to use the codecs module to encode as hex.

However, hex encoding is doubling the size of the input, so that's not really what you want if the goal is to save space. Isn't urlencoding (which you're doing below) already sufficient?

Anyway, we're using requests now, which can handle binary POST, so there's no need for hex encoding or urlencoding. Hopefully php can also handle that. I guess it also assumes UTF-8 encoding?

# Set flag.
post_body_data["c"] = 1

post_body = urllib.urlencode(post_body_data)

# logged before apikey added for security
self._log.info("sending: " + post_url + "E-M-O-N-C-M-S-A-P-I-K-E-Y&" + post_body)
Expand All @@ -70,6 +82,7 @@ def _process_post(self, databuffer):
# adopted

reply = self._send_post(post_url, post_body)

if reply == 'ok':
self._log.debug("acknowledged receipt with '" + reply + "' from " + self._settings['url'])
return True
Expand Down Expand Up @@ -138,5 +151,13 @@ def set(self, **kwargs):
self._log.info("Setting " + self.name + " sendstatus: " + setting)
self._settings[key] = setting
continue
elif key == 'compress':
self._log.info("Setting " + self.name + " compress: " + setting)
if setting == "1":
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

self._cms_settings['compress'] is 1, an int but this code expects the string "1".

setting = True
else:
setting = False
self._settings[key] = setting
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These 5 lines could just be
self._settings[key] = bool(setting)
if setting is indeed an int, but I think it will be a string in general so probably
self._settings[key] = bool(int(setting))

continue
else:
self._log.warning("'%s' is not valid for %s: %s" % (setting, self.name, key))