Handle Panorama stampede on restart after not running

[earthgecko]

Although Panorama is set up to record anomalies with the actual data in the
check file, similar to a queue, if Panorama happens to stall and is restarted
later the Panorama anomaly submissions are somewhat skewed and as it populates
the DB, it will probably fire any Skyline Analyzer or Mirage alerts that are on
any mysql metrics namespaces :)

Although it is creating a Panorama key, the keys will not match the event if
these are old check files, but realistically we still want to record the
anomalies in the DB with the same logical that is applied to current data, in
terms of setting keys or other similar method that will achieve the desired goal.

However that is not a trivial as it may seem I do not think.

Current workaround

Move the check files or let them be processed.

If they are processed, they will skew other metrics, even Skyline metrics and
possibly any related MySQL metrics too.

# Move the check files
PANORAMA_DIR="<YOUR_SKYLINE_PANORAMA_DIR>"  # e.g. PANORAMA_DIR="/opt/skyline/panaroma"

# STOP your Panorama if it is running

mkdir -p "${PANORAMA_DIR}/manually_skipped"
for i in $(find "${PANORAMA_DIR}/check" -type f -name "*.txt")
do
  /bin/mv -f "$i" "${PANORAMA_DIR}/manually_skipped/"
done

# START your Panorama, sorry some anomalies are not recorded

[earthgecko]

An additional variable was added to settings.py in v1.0.8 of PANORAMA_CHECK_MAX_AGE which allows for the discarding of any checks older than this value. This stopped the stampeding of MySQL. Although not ideal as anomalies are not recorded, however matching expiration times, etc on old metric checks and replaying the metric checks in the same fashion as realtime is non-trivial, so for now this will do.

PANORAMA_CHECK_MAX_AGE = 300
"""
:var PANORAMA_CHECK_MAX_AGE: Panorama will only process a check file if it is
    not older than PANORAMA_CHECK_MAX_AGE seconds.  If it is set to 0 it does
    all.  This setting just ensures if Panorama stalls for some hours and is
    restarted, the user can choose to discard older checks and miss anomalies
    being recorded if they so choose to, to prevent Panorama stampeding against
    MySQL if something went down and Panorama comes back online with lots of
    checks.
:vartype PANORAMA_CHECK_MAX_AGE: int
"""