New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export survey module data #26

Merged
merged 42 commits into from Jul 26, 2016

Conversation

Projects
None yet
2 participants
@mitar
Member

mitar commented Jul 22, 2016

A followup from #23 with branch made in this repository.

CdavM added some commits Jul 6, 2016

Changed parameter input from timestamp to datetime object.
Also fixed a bug that did not limit the end of a database query.
@mitar

This comment has been minimized.

Show comment
Hide comment
@mitar

mitar Jul 22, 2016

Member

@CdavM: Please test if it works. I have not run the code with my changes. If you notice anything, feel free to just fix it.

Member

mitar commented Jul 22, 2016

@CdavM: Please test if it works. I have not run the code with my changes. If you notice anything, feel free to just fix it.

@CdavM

This comment has been minimized.

Show comment
Hide comment
@CdavM

CdavM Jul 25, 2016

Contributor

@mitar : The code doesn't work anymore :(
When I try running your command to export the data, I get the following error:

Traceback (most recent call last):
  File "manage.py", line 9, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 345, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 348, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 399, in execute
    output = self.handle(*args, **options)
  File "/code/nodewatcher/modules/monitor/http/survey/management/commands/export_survey_data.py", line 68, in handle
    json.dump(graph, f, ensure_ascii=False, cls=django_json.DjangoJSONEncoder)
  File "/usr/lib/python2.7/json/__init__.py", line 190, in dump
    fp.write(chunk)
TypeError: must be unicode, not str

After reverting to f.write(json.dumps(...)), I get the following error:

Traceback (most recent call last):
  File "manage.py", line 9, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 345, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 348, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 399, in execute
    output = self.handle(*args, **options)
  File "/code/nodewatcher/modules/monitor/http/survey/management/commands/export_survey_data.py", line 68, in handle
    f.write(json.dumps(graph, ensure_ascii=False))
  File "/usr/lib/python2.7/json/__init__.py", line 250, in dumps
    sort_keys=sort_keys, **kw).encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python2.7/json/encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: datetime.datetime(2016, 7, 25, 4, 7, 48, tzinfo=<bson.tz_util.FixedOffset object at 0x7fb1a377bf90>) is not JSON serializable

So I'd just go back to my way of exporting. I have no idea how to track down these unicode errors. Or do you have any ideas on fixing this?

Contributor

CdavM commented Jul 25, 2016

@mitar : The code doesn't work anymore :(
When I try running your command to export the data, I get the following error:

Traceback (most recent call last):
  File "manage.py", line 9, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 345, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 348, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 399, in execute
    output = self.handle(*args, **options)
  File "/code/nodewatcher/modules/monitor/http/survey/management/commands/export_survey_data.py", line 68, in handle
    json.dump(graph, f, ensure_ascii=False, cls=django_json.DjangoJSONEncoder)
  File "/usr/lib/python2.7/json/__init__.py", line 190, in dump
    fp.write(chunk)
TypeError: must be unicode, not str

After reverting to f.write(json.dumps(...)), I get the following error:

Traceback (most recent call last):
  File "manage.py", line 9, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 353, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/__init__.py", line 345, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 348, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python2.7/dist-packages/django/core/management/base.py", line 399, in execute
    output = self.handle(*args, **options)
  File "/code/nodewatcher/modules/monitor/http/survey/management/commands/export_survey_data.py", line 68, in handle
    f.write(json.dumps(graph, ensure_ascii=False))
  File "/usr/lib/python2.7/json/__init__.py", line 250, in dumps
    sort_keys=sort_keys, **kw).encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
    return _iterencode(o, 0)
  File "/usr/lib/python2.7/json/encoder.py", line 184, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: datetime.datetime(2016, 7, 25, 4, 7, 48, tzinfo=<bson.tz_util.FixedOffset object at 0x7fb1a377bf90>) is not JSON serializable

So I'd just go back to my way of exporting. I have no idea how to track down these unicode errors. Or do you have any ideas on fixing this?

@CdavM

This comment has been minimized.

Show comment
Hide comment
@CdavM

CdavM Jul 26, 2016

Contributor

OK I fixed writing the output and I reverted to using 'v' and 'e' as keys, as that is done elsewhere in the code.

Contributor

CdavM commented Jul 26, 2016

OK I fixed writing the output and I reverted to using 'v' and 'e' as keys, as that is done elsewhere in the code.

@mitar

This comment has been minimized.

Show comment
Hide comment
@mitar

mitar Jul 26, 2016

Member

Above you mentioned that you had issues exporting dates, but then I do not see that you reverted to your old way of exporting dates as strings. So it works for you as it is now?

Member

mitar commented Jul 26, 2016

Above you mentioned that you had issues exporting dates, but then I do not see that you reverted to your old way of exporting dates as strings. So it works for you as it is now?

@mitar mitar merged commit 1b33bbf into development Jul 26, 2016

0 of 2 checks passed

continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
continuous-integration/travis-ci/push The Travis CI build is in progress
Details

@mitar mitar deleted the export-survey-module-data branch Jul 26, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment