Feature/ct 194/memcached uploader urls #192

lawschlosser · 2018-04-13T09:51:55Z

No description provided.

WMDConductor

LGTM - with small exception catching nit + Codacy considerations assumed sorted out.

WMDConductor · 2018-04-13T15:23:19Z

conductor/lib/common.py

+    assert type(char_count) == int, "char_count must be int. Got: %r (%s)" % (char_count, type(char_count))
+    try:
+        s_str = pformat(object_) if pretty else str(object_)
+    except:


Seems like we could make this: catch explicit:

except (ValueError, TypeError):

flebel · 2018-04-13T18:07:11Z

conductor/lib/common.py

@@ -7,6 +7,7 @@
 import multiprocessing
 import os
 import platform
+from pprint import pformat


from x import y go after import x statements.

eg. this is what isort gives me:

import base64 import datetime import functools import hashlib import json import logging import multiprocessing import os import platform import random import signal import subprocess import sys import time import traceback from pprint import pformat import yaml

We should probably decide as a team on how we want to sort imports. I haven't seen much in google style guide or pep8 (aside from grouping imports by builtin, thirdparty, internal).
Personally, I sort imports alphabetically, considering the uppermost namespace (parent packages) when evaluating order. I don't distinguish/consider whether it's a from vs import. I find that it makes it easier for me to look something up, if there is one continuous alphabetical ordering, rather than having multiple sections of their own ordering. But we're all weird humans, thinking in weird ways.

also, I noticed that isort has several different options that make dramatic differences (--no-sections, --order-by-type, --project, --thirdparty , etc). Might be worth playing around with these to find a workflow something that we all like (dislike ;) ).

Good call, let's make this a team decision.

flebel · 2018-04-13T18:08:17Z

conductor/lib/common.py

+    '''
+    Return a string representation of the given object, shortened to the given
+    char_count. This can be useful when printing/logging out data for debugging
+    purposes, but don't want an overwhelming wall of text to troll through.


s/troll/scroll/

That was the intended spelling. But after reading this: https://english.stackexchange.com/questions/122627/trawling-through-or-trolling-through
....I'm wondering if I should just change it to your recommendation (English is so hard!)

flebel · 2018-04-13T18:10:33Z

conductor/lib/uploader.py

+            try:
+                logger.info(self.manager.worker_queue_status_text())
+                logger.info(self.upload_status_text())
+            except Exception:


except: would also work as a catch-all.

I've been going back and forth on this recently (due to recent revelations that I'm somewhat embarrassed about). And coincidentally, this module makes for the perfect example!
As much as I have loved to catch all exceptions by simply not listing any specific types (when deemed appropriate), I didn't realize there was a subtle difference in doing so vs explicitly catching the Exception class. See https://stackoverflow.com/questions/730764/how-to-properly-ignore-exceptions

Usually it doesn't matter if we "catch all" exception, but because this module's application (the uploader), is a long running (daemon) process, that can only be stopped via killing it (ctrl-c, sigint, sigterm, etc), we have to be thoughtful to not disrupt any logic in the uploader that is expecting to encounter ctrl-c (KeyboardInterrupt).
Here's a great example of how you can unintentionally make trouble : https://fman.io/blog/an-inescapable-while-loop-in-python/

side note: here's an interesting diagram/hierarchy of pythons exception classes (as well as general good reading): https://airbrake.io/blog/python-exception-handling/class-hierarchy

flebel · 2018-04-13T18:41:46Z

conductor/lib/worker.py

    def stop_work(self):
        global WORKING
        WORKING = False  # stop any new jobs from being created
        self.drain_queues()  # clear out any jobs in queue
        self.kill_workers()  # kill all threads
+        self.kill_metricstore()


Can you rename the method to kill_metric_store for consistency with the above variable self.metric_store?

sure, makes sense

flebel · 2018-04-13T18:42:19Z

conductor/lib/worker.py

+        return (queue.get_nowait())
+    except Queue.Empty:
+        return
+    except Exception:


This can be simplified to except:.

#192 (comment)

flebel · 2018-04-16T18:18:39Z

conductor/lib/common.py

@@ -7,6 +7,7 @@
 import multiprocessing
 import os
 import platform
+from pprint import pformat


Good call, let's make this a team decision.

- Now properly terminates the following threads after uploading a job: ErrorThread MetricStore PrintStatusThread ReporterThread - Disclaimer: this code really stinks. Needs an entire re-write.

- Activated for POST /jobs/

lawschlosser requested review from flebel, easythrees and WMDConductor April 13, 2018 09:51

WMDConductor approved these changes Apr 13, 2018

View reviewed changes

flebel approved these changes Apr 13, 2018

View reviewed changes

lawschlosser force-pushed the feature/CT-194/memcached-uploader-urls branch from 857329c to fc0766e Compare April 13, 2018 23:00

AtomicConductor deleted a comment Apr 13, 2018

lawschlosser force-pushed the feature/CT-194/memcached-uploader-urls branch from fc0766e to 4a1f334 Compare April 14, 2018 03:22

AtomicConductor deleted a comment Apr 14, 2018

flebel approved these changes Apr 16, 2018

View reviewed changes

lawschlosser added 8 commits April 25, 2018 13:04

CT-187 Fix for thread termination issues

5d14837

- Now properly terminates the following threads after uploading a job: ErrorThread MetricStore PrintStatusThread ReporterThread - Disclaimer: this code really stinks. Needs an entire re-write.

chore: converted list to iterator for performance considerations

adec441

uploader args fix

e46501e

fix for hanging thread when error occurs

e4caea2

misc uploader improvements

1191f35

CT-194 added memcache to uploader

6bae665

CT-171 Added compression option for api make_request

8587159

- Activated for POST /jobs/

Increased uploader MD5OutputWorker batch size from 20 to 100

03a1f1d

lawschlosser force-pushed the feature/CT-194/memcached-uploader-urls branch from 4a1f334 to 03a1f1d Compare April 25, 2018 20:05

AtomicConductor deleted a comment Apr 25, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/ct 194/memcached uploader urls #192

Feature/ct 194/memcached uploader urls #192

lawschlosser commented Apr 13, 2018

WMDConductor left a comment

WMDConductor Apr 13, 2018

flebel Apr 13, 2018

lawschlosser Apr 13, 2018

flebel Apr 16, 2018

flebel Apr 13, 2018

lawschlosser Apr 13, 2018

flebel Apr 13, 2018

lawschlosser Apr 13, 2018

flebel Apr 13, 2018

lawschlosser Apr 13, 2018

flebel Apr 13, 2018

lawschlosser Apr 13, 2018

flebel Apr 16, 2018

Feature/ct 194/memcached uploader urls #192

Are you sure you want to change the base?

Feature/ct 194/memcached uploader urls #192

Conversation

lawschlosser commented Apr 13, 2018

WMDConductor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment