Merge pull request #136 from pliablepixels/pyzmutils

migrate to pyzm logger

docs/guides/breaking.rst

@@ -1,6 +1,14 @@
 Breaking Changes
 ----------------
 
+Version 4.0 onwards
+~~~~~~~~~~~~~~~~~~~~
+- Hook versions will now always be ``<ES version>.x``, so in this case ``4.0.1``
+- Hooks have now migrated to using a `proper python ZM logger module <https://pypi.org/project/pyzmutils/>`__ so it better integrates with ZM logging 
+- To view detection logs, you now need to follow the standard ZM logging process. See :ref:`hooks-logging` documentation for more details)
+- You no longer have to manually install python requirements, the setup process should automatically install them
+
+
 Version 3.9 onwards
 ~~~~~~~~~~~~~~~~~~~~
 - Hooks now add ALPR, so you need to run `sudo -H pip install -r requirements.txt` again

docs/guides/hooks.rst

@@ -8,19 +8,16 @@ Machine Learning Hooks
 
 .. important::
 
-        Please don't ask me basic questions like "pip command not found" or
+        Please don't ask me basic questions like "pip3 command not found" or
         "cv2 not found" - what do I do? Hooks require some terminal
         knowledge and familiarity with troubleshooting. I don't plan to
         provide support for these hooks. They are for reference only
 
 Limitations
 ~~~~~~~~~~~
 
--  Only tested with ZM 1.32+. May or may not work with older versions
--  Needs Python3 (I used to support Python2, but not any more)
-   Python2 will be deprecated in 2020. May as well update.
-- If you are using Python3, you can use ``python`` and ``pip``. They will point to python3 versions. You typically need to use ``python3`` and ``pip3`` when you have a mixed Python 2.x and 3.x install
-
+- Only tested with ZM 1.32+. May or may not work with older versions
+- Needs Python3 (I used to support Python2, but not any more). Python2 will be deprecated in 2020. May as well update.
 
 What
 ~~~~
@@ -75,6 +72,7 @@ automatically, especially if you don't need face recognition.
 Note, if you installed ``face_recognition`` without blas, do this:
 
 ::
+
         sudo -H pip3 uninstall dlib
         sudo -H pip3 uninstall face-recognition
         sudo apt-get install libopenblas-dev liblapack-dev libblas-dev # this is the important part
@@ -216,17 +214,41 @@ If it doesn't work, go back and figure out where you have a problem
       -  If you are running ZM >= 1.33, you can use all fid modes
          without requiring to enable frames in storage
 
+
+.. _hooks-logging:
+
+Logging
+~~~~~~~~~
+
+Starting version 4.0.x, the hooks now use ZM logging, thanks to a `python wrapper <https://pypi.org/project/pyzmutils/>`__ I wrote recently that taps into ZM's logging system. This also means it is no longer as easy as enabling ``log_level=debug`` in ``objdetect.ini``. Infact, that option has been removed. Follow standard ZM logging options for the hooks. Here is what I do:
+
+- In ``ZM->Options->Logs:``
+
+  - LOG_LEVEL_FILE = debug
+  - LOG_LEVEL_SYSLOG = Info
+  - LOG_LEVEL_DATABASE = Info
+  - LOG_DEBUG is on
+  - LOG_DEBUG_TARGET = ``_zmesdetect`` (if you have other targets, just separate them with ``|`` - example, ``_zmc|_zmesdetect``)
+
+  The above config. will store debug logs in my ``/var/log/zm`` directory, while Info level logs will be recorded in syslog and DB.
+
+  So now, to view hooks/detect logs, all I do is:
+
+  ::
+
+    tail -f  /var/log/zm/zmesdetect*.log
+
+  Note that the detection code registers itself as ``zmesdetect`` with ZM. When it is invoked with a specific monitor ID (usually the case), then the component is named ``zmesdetect_mX.log`` where ``X`` is the monitor ID. In other words, that now gives you one log per monitor (just like ``/var/log/zm/zmc_mX.log``) which makes it easy to debug/isolate. 
+
 Troubleshooting
 ~~~~~~~~~~~~~~~
 
 -  In general, I expect you to debug properly. Please don't ask me basic
    questions without investigating logs yourself
 -  Always run ``detect_wrapper.sh`` in manual mode first to make sure it
    works
--  Set ``log_level`` to ``debug`` in ``objdetect.ini`` before you run it
-   manually
--  When you run in manually, view system logs in another window to look
-   for errors (``tail -f /var/log/syslog | grep yolo``)
+-  To get debug logs, Make sure your ``LOG_DEBUG`` in ZM Options->Logs is set to on and your ``LOG_DEBUG_TARGET`` option includes ``_zmesdetect`` (or is empty)
+-  You can view debug logs for detection by doing ``tail -f  /var/log/zm/zmesdetect*.log``
 -  One of the big reasons why object detection fails is because the hook
    is not able to download the image to check. This may be because your
    ZM version is old or other errors. Some common issues:
@@ -270,27 +292,58 @@ instead of full yolo weights. Again, please readd the comments in
 How to use license plate recognition
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-I use `Plate Recognizer <https://platerecognizer.com>`__ for license plate recognition. It uses a deep learning model that does a far better job than OpenALPR (based on my tests). The class is abstracted, obviously, so in future I may add local models. For now, you will have to get a license key from them (they have a `free tier <https://platerecognizer.com/pricing/>`__ that allows 2500 lookups per month)
+Two ALPR options are provided: 
+
+- `Plate Recognizer <https://platerecognizer.com>`__ . It uses a deep learning model that does a far better job than OpenALPR (based on my tests). The class is abstracted, obviously, so in future I may add local models. For now, you will have to get a license key from them (they have a `free tier <https://platerecognizer.com/pricing/>`__ that allows 2500 lookups per month)
+- `OpenALPR <https://www.openalpr.com>`__ . While OpenALPR's detection is not as good as Plate Recognizer, when it does detect, it provides a lot more information (like car make/model/year etc.)
 
 To enable alpr, simple add `alpr` to `models`. You will also have to add your license key to the ``[alpr]`` section of ``objdetect.ini``
 
-Note that since this is a remote service hosted by a 3rd party, you can't use ALPR without an internet connection. 
+This is an example config that uses plate recognizer:
 
 ::
 
   models = yolo,alpr
 
   [alpr]
   alpr_service=plate_recognizer
-  alpr_key=<the key>
-  alpr_use_after_detection_only=yes
+  # If you want to host a local SDK https://app.platerecognizer.com/sdk/
+  #alpr_url=https://localhost:8080
+  # Plate recog replace with your api key
+  alpr_key=KEY
+  # if yes, then it will log usage statistics of the ALPR service
+  platerec_stats=no
+  # If you want to specify regions. See http://docs.platerecognizer.com/#regions-supported
+  #platerec_regions=['us','cn','kr']
+  # minimal confidence for actually detecting a plate
+  platerec_min_dscore=0.1
+  # minimal confidence for the translated text
+  platerec_min_score=0.2
+
+
+This is an example config that uses OpenALPR:
+
+::
+
+  models = yolo,alpr
+
+  [alpr]
+  alpr_service=open_alpr
+  alpr_key=SECRET
+
+  # For an explanation of params, see http://doc.openalpr.com/api/?api=cloudapi
+  openalpr_recognize_vehicle=1
+  openalpr_country=us
+  openalpr_state=ca
+  openalpr returns percents, but we convert to between 0 and 1
+  openalpr_min_confidence=0.3
 
-Leave ``alpr_service`` and ``alpr_use_after_detection_only`` to the default values. 
+Leave ``alpr_use_after_detection_only`` to the default values. 
 
 How license plate recognition will work
 ''''''''''''''''''''''''''''''''''''''''
 
-- To save on platerecognizer API calls, the code will only invoke remote APIs if a vehicle is detected
+- To save on  API calls, the code will only invoke remote APIs if a vehicle is detected
 - This also means you MUST specify yolo along with alpr
 
 
@@ -337,7 +390,7 @@ known faces images
 
 -  Only put in one image per person
 -  Make sure the face is recognizable
--  crop it to around 200 pixels width (doesn't seem to need bigger
+-  crop it to around 400 pixels width (doesn't seem to need bigger
    images, but experiment. Larger the image, the larger the memory
    requirements)
 

hook/detect.py

@@ -53,9 +53,9 @@ def append_suffix(filename, token):
 args = vars(args)
 
 if args['monitorid']:
-    log.init('detect',args['monitorid'])
+    log.init(process_name='zmesdetect_'+'m'+args['monitorid'])
 else:
-    log.init('detect')
+    log.init(process_name='zmesdetect')
 
 g.logger.info ('---------| app version: {} |------------'.format(__version__))
 if args['version']:
@@ -203,7 +203,7 @@ def append_suffix(filename, token):
 
         # now filter these with polygon areas
         #g.logger.debug ("INTERIM BOX = {} {}".format(b,l))
-        b, l, c = img.processIntersection(b, l, c, match)
+        b, l, c = img.processFilters(b, l, c, match)
         if use_alpr:
             vehicle_labels = ['car','motorbike', 'bus','truck', 'boat']
             if not set(l).isdisjoint(vehicle_labels) or try_next_image: 
@@ -244,7 +244,7 @@ def append_suffix(filename, token):
                         })
                     # Now add plate objects
                     for i, al in enumerate(alpr_l):
-                        g.logger.debug ('ALPR Found {} at {} with score:{}'.format(al, alpr_b[i], alpr_c[i]))
+                        g.logger.info ('ALPR Found {} at {} with score:{}'.format(al, alpr_b[i], alpr_c[i]))
                         b.append(alpr_b[i])
                         l.append(al)
                         c.append(alpr_c[i])
@@ -264,7 +264,7 @@ def append_suffix(filename, token):
                     saved_file = filename
                     try_next_image = True
                 else: # no plates, no more to try
-                    g.logger.debug ('We did not find license plates, and there are no more images to try')
+                    g.logger.info ('We did not find license plates, and there are no more images to try')
                     if saved_bbox:
                         g.logger.debug ('Going back to matches in first image')
                         b = saved_bbox
@@ -330,7 +330,7 @@ def append_suffix(filename, token):
                 label.extend(l)
                 conf.extend(c)
                 classes.append(m.get_classes())
-                g.logger.debug('labels found: {}'.format(l))
+                g.logger.info('labels found: {}'.format(l))
                 g.logger.debug ('match found in {}, breaking file loop...'.format(filename))
                 matched_file = filename
                 break # if we found a match, no need to process the next file
@@ -349,7 +349,7 @@ def append_suffix(filename, token):
 #g.logger.debug ('FINAL LIST={} AND {}'.format(bbox,label))
 
 if not matched_file:
-        g.logger.debug('No patterns found using any models in all files')
+        g.logger.info('No patterns found using any models in all files')
 
 else:
     # we have matches
@@ -362,7 +362,8 @@ def append_suffix(filename, token):
 
     #for idx, b in enumerate(bbox):
     #g.logger.debug ("DRAWING {}".format(b))
-    out = img.draw_bbox(image, bbox, label, classes, conf, None, False)
+
+    out = img.draw_bbox(image, bbox, label, classes, conf, None, g.config['show_percent']=='yes')
     image = out
 
     if g.config['frame_id'] == 'bestmatch':
@@ -401,7 +402,7 @@ def append_suffix(filename, token):
 
     if g.config['match_past_detections'] == 'yes' and args['monitorid']:
             # point detections to post processed data set
-            g.logger.debug ('Removing matches to past detections')
+            g.logger.info ('Removing matches to past detections')
             bbox_t, label_t, conf_t = img.processPastDetection(bbox, label, conf, args['monitorid'])   
             # save current objects for future comparisons
             g.logger.debug ('Saving detections for monitor {} for future match'.format(args['monitorid']))
@@ -429,7 +430,7 @@ def append_suffix(filename, token):
     if pred != '':
         pred = pred.rstrip(',')
         pred = prefix + 'detected:' + pred
-        g.logger.debug('Prediction string:{}'.format(pred))
+        g.logger.info('Prediction string:{}'.format(pred))
         print (pred)
 
     # end of matched_file

hook/objectconfig.ini

@@ -5,11 +5,10 @@
 # duplicate it inside the correct [monitor-<num>] section
 
 [general]
-portal=https://server/zm
+portal=https:/server/zm
 user=admin
 password=password
 allow_self_signed=yes
-
 # if yes, last detection will be stored for monitors
 # and bounding boxes that match, along with labels
 # will be discarded for new detections. This may be helpful
@@ -66,9 +65,6 @@ write_image_to_zm=yes
 # hog/face shows 100% always
 show_percent=no
 
-# log level can be info,error,debug
-log_level=debug
-
 # color to be used to draw the polygons you specified
 poly_color=(255,255,255)
 #import_zm_zones=no
@@ -91,6 +87,7 @@ poly_color=(255,255,255)
 [monitor-8]
 # my driveway
 detect_pattern=(person|car|motorbike|bus|truck|boat)
+delete_after_analyze=no
 #detect_pattern=.*
 #import_zm_zones=yes
 my_driveway_perimeter=306,356 1003,341 1074,683 154,715
@@ -111,7 +108,7 @@ models=yolo,alpr
 # the characters in front implement what is 
 # called a negative look ahead
 
-detect_pattern=^(?!potted plant|pottedplant|bench)
+detect_pattern=^(?!potted plant|pottedplant|bench|broccoli)
 #detect_pattern=.*
 
 # local model overrides global
@@ -168,6 +165,7 @@ face_model=cnn
 [yolo]
 yolo_type=full
 #yolo_type=tiny
+yolo_min_confidence=0.5
 
 config=/var/lib/zmeventnotification/models/yolov3/yolov3.cfg
 weights=/var/lib/zmeventnotification/models/yolov3/yolov3.weights
@@ -205,11 +203,11 @@ face_upsample_times=1
 alpr_use_after_detection_only=yes
 
 # -----| If you are using plate recognizer | ------
-#alpr_service=plate_recognizer
+alpr_service=plate_recognizer
 # If you want to host a local SDK https://app.platerecognizer.com/sdk/
 #alpr_url=https://localhost:8080
 # Plate recog replace with your api key
-#alpr_key=__PLATERECOG_KEY__
+alpr_key=KEY
 # if yes, then it will log usage statistics of the ALPR service
 platerec_stats=no
 # If you want to specify regions. See http://docs.platerecognizer.com/#regions-supported
@@ -221,13 +219,13 @@ platerec_min_score=0.2
 
 
 # ----| If you are using openALPR |-----
-alpr_service=open_alpr
-alpr_key=__OPENALPR_SECRET__
+#alpr_service=open_alpr
+#alpr_key=SECRET
 
 # For an explanation of params, see http://doc.openalpr.com/api/?api=cloudapi
-openalpr_recognize_vehicle=1
+#openalpr_recognize_vehicle=1
 #openalpr_country=us
 #openalpr_state=ca
 # openalpr returns percents, but we convert to between 0 and 1
-openalpr_min_confidence=0.3
+#openalpr_min_confidence=0.3
 

hook/setup.py

@@ -14,7 +14,7 @@
 AUTHOR_EMAIL = 'pliablepixels@gmail.com'
 AUTHOR = 'Pliable Pixels'
 LICENSE = 'GPL'
-INSTALL_REQUIRES=['opencv_contrib_python', 'numpy', 'requests', 'Shapely', 'imutils']
+INSTALL_REQUIRES=['opencv_contrib_python', 'numpy', 'requests', 'Shapely', 'imutils', 'pyzmutils']
 
 
 here = os.path.abspath(os.path.dirname(__file__))

hook/zmes_hook_helpers/__init__.py

@@ -1,2 +1,2 @@
-__version__ = "3.3.6"
+__version__ = "4.0.2"
 VERSION=__version__

hook/zmes_hook_helpers/common_params.py

@@ -75,11 +75,6 @@
             'default': 'no',
             'type': 'string'
         },
-        'log_level':{
-            'section': 'general',
-            'default': 'info',
-            'type': 'string'
-        },
         'allow_self_signed':{
             'section': 'general',
             'default': 'yes',
@@ -153,6 +148,12 @@
             'default': '/var/lib/zmeventnotification/models/tinyyolo/yolov3-tiny.txt',
             'type': 'string'
         },
+
+        'yolo_min_confidence': {
+            'section': 'yolo',
+            'default': '0.4',
+            'type': 'float'
+        },
 
         # HOG
         'stride':{