Add duplicate checking and limited matching optimizations and other

scheduler related changes.

The three major changes are as follows.

Split the checks against the oldrecorded and recorded tables for
previous recordings away from the "place" phase of scheduling and call
it the "check" phase.  This makes it easier to report how much time is
spent doing this and leads to the next major change.

Drastically reduce the number of checks against the oldrecorded and
recorded tables for previous recordings.  Historically, this has been
the most significant contributor to long scheduler runs.  Not every
unnecessary check is eliminated, but the vast majority of them are.
Trying to remove the few remaining ones would probably take longer
than simply rechecking them.

Allow for checking a subset of the program table when new guide data
is available.  This is primarily aimed at EIT scanning where typically
only a few hours of guide data for specific channels is updated in an
orderly fashion.  Not rechecking the unchanged guide data greatly
reduces the amount of work the scheduler has to do.  Please note the
EIT scanner has not been updated yet to take advantage of this change.

The other changes are as follows.

Use more expressive reschedule requests.  This is primarily to support
the changes listed above.  It also makes it easier to understand why
reschedules occurred when reading the logs and might lead to the
elimination of unnecessary reschedules in the future.

  MATCH reschedule requests should be used when the guide data or a
  specific recording rule is changed.  The syntax is as follows.

  RESCHEDULE_RECORDINGS
  MATCH <recordid> <sourceid> <mplexid> <maxstarttime> <reason>

  maxstarttime should be in ISO format.  If recordid, sourceid or
  mplexid are non-zero or maxstarttime is a valid time, the scheduler
  will restrict itself to recording rules and guide data matching
  those parameters.  reason is purely for purposes of logging.

  CHECK reschedule requests should be used when the status of a
  specific episode is affected such as when "never record" or "allow
  re-record" are selected or a recording finishes or is deleted.  The
  syntax is as follows.

  RESCHEDULE_RECORDINGS
  CHECK <recstatus> <recordid> <findid> <reason>
  <title>
  <subtitle>
  <description>
  <programid>

  recordid should be the parent recordid, when applicable, otherwise,
  the normal recordid.  Setting programid to '**any**" and title to ""
  is a special case used to emulate an old reschedule request for
  recordid 0.  Setting programid to "**any**" and title to other than
  "" is another special case used when all entries for a title are
  deleted from oldrecorded.  recstatus and reason are purely for
  purposes of logging.

  PLACE reschedule request should be used in all other cases.  The
  syntax is as follows.

  RESCHEDULE_RECORDINGS
  PLACE <reason>

  reason is purely for purposes of logging.

Update the PHP and Python bindings to use the new reschedule requests.
Please note the bindings API has not changed to make full use of the
new requests.  I'm leaving it to the bindings maintainers to decide
how best to do that.

Clear record.duplicate when a recording is queued for deletion.  It
should have been this way all along and avoids a redundant reschedule
when the file is actually unlinked.

Lower the very detailed scheduler placement logging to LOG_DEBUG.
This makes VB_SCHEDULE/LOG_INFO more useful for less voluminous
purposes.

Add findid to the recordmatch table.  This avoids having to calculate
it in multiple places.

Tighten the window for rescheduling long running programs from "24
hours ago" to "~8 hours ago".

Avoid creating unnecessary RecordingInfo objects that are going to get
deleted later anyway.

Fixes #10533

Author: gigem

mythtv/bindings/perl/MythTV.pm

@@ -106,15 +106,15 @@ package MythTV;
 # Note: as of July 21, 2010, this is actually a string, to account for proto
 # versions of the form "58a".  This will get used if protocol versions are 
 # changed on a fixes branch ongoing.
-    our $PROTO_VERSION = "72";
-    our $PROTO_TOKEN = "D78EFD6F";
+    our $PROTO_VERSION = "73";
+    our $PROTO_TOKEN = "D7FE8D6F";
 
 # currentDatabaseVersion is defined in libmythtv in
 # mythtv/libs/libmythtv/dbcheck.cpp and should be the current MythTV core
 # schema version supported in the main code.  We need to check that the schema
 # version in the database is as expected by the bindings, which are expected
 # to be kept in sync with the main code.
-    our $SCHEMA_VERSION = "1299";
+    our $SCHEMA_VERSION = "1300";
 
 # NUMPROGRAMLINES is defined in mythtv/libs/libmythtv/programinfo.h and is
 # the number of items in a ProgramInfo QStringList group used by

mythtv/bindings/php/MythBackend.php

@@ -11,8 +11,8 @@ class MythBackend {
 
 // MYTH_PROTO_VERSION is defined in libmyth in mythtv/libs/libmyth/mythcontext.h
 // and should be the current MythTV protocol version.
-    static $protocol_version        = '72';
-    static $protocol_token          = 'D78EFD6F';
+    static $protocol_version        = '73';
+    static $protocol_token          = 'D7FE8D6F';
 
 // The character string used by the backend to separate records
     static $backend_separator       = '[]:[]';
@@ -205,7 +205,17 @@ public function queryProgramRows($query = null, $offset = 1) {
  * need to indicate every record rule is affected, then use -1.
 /**/
     public function rescheduleRecording($recordid = -1) {
-        $this->sendCommand('RESCHEDULE_RECORDINGS '.$recordid);
+        if ($recordid == 0) {
+            $this->sendCommand(array('RESCHEDULE_RECORDINGS ', 
+                                     'CHECK 0 0 0 PHP',
+                                     '', '', '', '**any**'));
+        }
+        else {
+            if ($recordid == -1)
+                $recordid = 0;
+            $this->sendCommand(array('RESCHEDULE_RECORDINGS ', 
+                                     'MATCH '.$recordid.' 0 0 - PHP'));
+        }
         Cache::clear();
         if ($this->listenForEvent('SCHEDULE_CHANGE'))
             return true;

mythtv/bindings/python/MythTV/mythproto.py

@@ -670,7 +670,17 @@ def reschedule(self, recordid=-1, wait=False):
                                     ['BACKEND_MESSAGE',
                                      'SCHEDULE_CHANGE',
                                      'empty']))
-        self.backendCommand('RESCHEDULE_RECORDINGS '+str(recordid))
+        if recordid == 0:
+            self.backendCommand(BACKEND_SEP.join(\
+                ['RESCHEDULE_RECORDINGS',
+                 'CHECK 0 0 0 Python',
+                 '', '', '', '**any**']))
+        else:
+            if recordid == -1:
+                recordid = 0
+            self.backendCommand(BACKEND_SEP.join(\
+                ['RESCHEDULE_RECORDINGS',
+                 'MATCH ' + str(recordid) + ' 0 0 - Python']))
         if wait:
             eventlock.wait()
 

mythtv/bindings/python/MythTV/static.py

@@ -5,11 +5,11 @@
 """
 
 OWN_VERSION = (0,25,-1,3)
-SCHEMA_VERSION = 1299
+SCHEMA_VERSION = 1300
 NVSCHEMA_VERSION = 1007
 MUSICSCHEMA_VERSION = 1018
-PROTO_VERSION = '72'
-PROTO_TOKEN = 'D78EFD6F'
+PROTO_VERSION = '73'
+PROTO_TOKEN = 'D7FE8D6F'
 BACKEND_SEP = '[]:[]'
 INSTALL_PREFIX = '/usr/local'
 

mythtv/libs/libmyth/programinfo.cpp

@@ -2678,7 +2678,8 @@ void ProgramInfo::SaveDeletePendingFlag(bool deleteFlag)
     MSqlQuery query(MSqlQuery::InitCon());
 
     query.prepare("UPDATE recorded"
-                  " SET deletepending = :DELETEFLAG"
+                  " SET deletepending = :DELETEFLAG, "
+                  "     duplicate = 0 "
                   " WHERE chanid = :CHANID"
                   " AND starttime = :STARTTIME ;");
     query.bindValue(":CHANID", chanid);

mythtv/libs/libmythbase/mythversion.h

@@ -12,7 +12,7 @@
 /// Update this whenever the plug-in API changes.
 /// Including changes in the libmythbase, libmyth, libmythtv, libmythav* and
 /// libmythui class methods used by plug-ins.
-#define MYTH_BINARY_VERSION "0.25.20120408-1"
+#define MYTH_BINARY_VERSION "0.26.20120411-1"
 
 /** \brief Increment this whenever the MythTV network protocol changes.
  *
@@ -35,8 +35,8 @@
  *       mythtv/bindings/python/MythTV/static.py (version number)
  *       mythtv/bindings/python/MythTV/mythproto.py (layout)
  */
-#define MYTH_PROTO_VERSION "72"
-#define MYTH_PROTO_TOKEN "D78EFD6F"
+#define MYTH_PROTO_VERSION "73"
+#define MYTH_PROTO_TOKEN "D7FE8D6F"
 
 /** \brief Increment this whenever the MythTV core database schema changes.
  *
@@ -57,7 +57,7 @@
  *      mythtv/bindings/php/MythBackend.php
 #endif
 
-#define MYTH_DATABASE_VERSION "1299"
+#define MYTH_DATABASE_VERSION "1300"
 
 
  MBASE_PUBLIC  const char *GetMythSourceVersion();

mythtv/libs/libmythtv/dbcheck.cpp

@@ -1948,6 +1948,18 @@ NULL
             return false;
     }
 
+    if (dbver == "1299")
+    {
+        const char *updates[] = {
+"ALTER TABLE recordmatch ADD COLUMN findid INT NOT NULL DEFAULT 0",
+"ALTER TABLE recordmatch ADD INDEX (recordid, findid)",
+NULL
+};
+
+        if (!performActualUpdate(updates, "1300", dbver))
+            return false;
+    }
+
     return true;
 }
 

mythtv/libs/libmythtv/eitscanner.cpp

@@ -177,7 +177,7 @@ void EITScanner::RescheduleRecordings(void)
         QDateTime::currentDateTime().addSecs(kMinRescheduleInterval);
     resched_lock.unlock();
 
-    ScheduledRecording::signalChange(-1);
+    ScheduledRecording::RescheduleMatch(0, 0, 0, QDateTime(), "EITScanner");
 }
 
 /** \fn EITScanner::StartPassiveScan(ChannelBase*, EITSource*, bool)

mythtv/libs/libmythtv/recordinginfo.cpp

@@ -1155,7 +1155,7 @@ void RecordingInfo::ReactivateRecording(void)
     if (!result.exec())
         MythDB::DBError("ReactivateRecording", result);
 
-    ScheduledRecording::signalChange(0);
+    ScheduledRecording::ReschedulePlace("Reactivate");
 }
 
 /**
@@ -1223,7 +1223,7 @@ void RecordingInfo::AddHistory(bool resched, bool forcedup, bool future)
     // The adding of an entry to oldrecorded may affect near-future
     // scheduling decisions, so recalculate if told
     if (resched)
-        ScheduledRecording::signalChange(0);
+        ScheduledRecording::RescheduleCheck(*this, "AddHistory");
 }
 
 /** \fn RecordingInfo::DeleteHistory(void)
@@ -1257,7 +1257,7 @@ void RecordingInfo::DeleteHistory(void)
 
     // The removal of an entry from oldrecorded may affect near-future
     // scheduling decisions, so recalculate
-    ScheduledRecording::signalChange(0);
+    ScheduledRecording::RescheduleCheck(*this, "DeleteHistory");
 }
 
 /** \fn RecordingInfo::ForgetHistory(void)
@@ -1321,7 +1321,7 @@ void RecordingInfo::ForgetHistory(void)
 
     // The removal of an entry from oldrecorded may affect near-future
     // scheduling decisions, so recalculate
-    ScheduledRecording::signalChange(0);
+    ScheduledRecording::RescheduleCheck(*this, "ForgetHistory");
 }
 
 /** \fn RecordingInfo::SetDupHistory(void)
@@ -1347,7 +1347,7 @@ void RecordingInfo::SetDupHistory(void)
     if (!result.exec())
         MythDB::DBError("setDupHistory", result);
 
-    ScheduledRecording::signalChange(0);
+    ScheduledRecording::RescheduleCheck(*this, "SetHistory");
 }
 
 /**

mythtv/libs/libmythtv/recordinginfo.h

@@ -228,8 +228,6 @@ class MTV_PUBLIC RecordingInfo : public ProgramInfo
     void ApplyTranscoderProfileChange(const QString &profile) const;//pi
     void ApplyTranscoderProfileChangeById(int);
 
-    static void signalChange(int recordid);
-
     RecStatusType oldrecstatus;
     RecStatusType savedrecstatus;
     bool future;

mythtv/libs/libmythtv/recordingrule.cpp

@@ -8,7 +8,7 @@
 #include "mythcorecontext.h"
 
 // libmythtv
-#include "scheduledrecording.h" // For signalChange()
+#include "scheduledrecording.h" // For RescheduleMatch()
 #include "playgroup.h" // For GetInitialName()
 #include "recordingprofile.h" // For constants
 #include "mythmiscutil.h"
@@ -380,7 +380,8 @@ bool RecordingRule::Save(bool sendSig)
         m_recordID = query.lastInsertId().toInt();
 
     if (sendSig)
-        ScheduledRecording::signalChange(m_recordID);
+        ScheduledRecording::RescheduleMatch(m_recordID, 0, 0, QDateTime(),
+            QString("SaveRule %1").arg(m_title));
 
     return true;
 }
@@ -408,7 +409,8 @@ bool RecordingRule::Delete(bool sendSig)
     }
 
     if (sendSig)
-        ScheduledRecording::signalChange(m_recordID);
+        ScheduledRecording::RescheduleMatch(m_recordID, 0, 0, QDateTime(),
+            QString("DeleteRule %1").arg(m_title));
 
     // Set m_recordID to zero, the rule is no longer in the database so it's
     // not valid. Should you want, this allows a rule to be removed from the

mythtv/libs/libmythtv/scheduledrecording.cpp

@@ -9,22 +9,56 @@ ScheduledRecording::~ScheduledRecording()
 {
 }
 
-void ScheduledRecording::signalChange(int recordid)
+void ScheduledRecording::SendReschedule(const QStringList &request)
 {
     if (gCoreContext->IsBackend())
     {
-        MythEvent me(QString("RESCHEDULE_RECORDINGS %1").arg(recordid));
+        MythEvent me(QString("RESCHEDULE_RECORDINGS"), request);
         gCoreContext->dispatch(me);
     }
     else
     {
         QStringList slist;
-        slist << QString("RESCHEDULE_RECORDINGS %1").arg(recordid);
+        slist << QString("RESCHEDULE_RECORDINGS");
+        slist << request;
         if (!gCoreContext->SendReceiveStringList(slist))
             LOG(VB_GENERAL, LOG_ERR,
-                QString("Error rescheduling id %1 in "
-                        "ScheduledRecording::signalChange") .arg(recordid));
+                QString("Error rescheduling %1 in "
+                        "ScheduledRecording::SendReschedule").arg(request[0]));
     }
 }
 
+QStringList ScheduledRecording::BuildMatchRequest(uint recordid, 
+                uint sourceid, uint mplexid, const QDateTime &maxstarttime, 
+                const QString &why)
+{
+    return QStringList(QString("MATCH %1 %2 %3 %4 %5")
+                       .arg(recordid).arg(sourceid).arg(mplexid)
+                       .arg(maxstarttime.isValid() ?
+                            maxstarttime.toString(Qt::ISODate) :
+                            "-")
+                       .arg(why));
+};
+
+QStringList ScheduledRecording::BuildCheckRequest(const RecordingInfo &recinfo,
+                                                  const QString &why)
+{
+    return QStringList(QString("CHECK %1 %2 %3 %4")
+                       .arg(recinfo.GetRecordingStatus())
+                       .arg(recinfo.GetParentRecordingRuleID() ?
+                            recinfo.GetParentRecordingRuleID() :
+                            recinfo.GetRecordingRuleID())
+                       .arg(recinfo.GetFindID())
+                       .arg(why))
+        << recinfo.GetTitle()
+        << recinfo.GetSubtitle()
+        << recinfo.GetDescription()
+        << recinfo.GetProgramID();
+};
+
+QStringList ScheduledRecording::BuildPlaceRequest(const QString &why)
+{
+    return QStringList(QString("PLACE %1").arg(why));
+};
+
 /* vim: set expandtab tabstop=4 shiftwidth=4: */

mythtv/libs/libmythtv/scheduledrecording.h

@@ -2,18 +2,43 @@
 #define SCHEDULEDRECORDING_H
 
 #include "mythtvexp.h"
+#include "qdatetime.h"
+#include "recordinginfo.h"
 
 class MTV_PUBLIC ScheduledRecording
 {
+    friend class Scheduler;
+
   public:
+    // Use when a recording rule or program data changes.  Use 0 for
+    // recordid when all recordids are potentially affected, Use
+    // invalid starttime and 0 for chanids when not time nor channel
+    // specific.
+    static void RescheduleMatch(uint recordid, uint sourceid, uint mplexid,
+                             const QDateTime &maxstarttime, const QString &why)
+        { SendReschedule(BuildMatchRequest(recordid, sourceid, mplexid,
+                                           maxstarttime, why)); };
+
+    // Use when previous or current recorded duplicate status changes.
+    static void RescheduleCheck(const RecordingInfo &recinfo, 
+                                const QString &why)
+        { SendReschedule(BuildCheckRequest(recinfo, why)); };
+
+    // Use when none of recording rule, program data or duplicate
+    // status changes.
+    static void ReschedulePlace(const QString &why)
+        { SendReschedule(BuildPlaceRequest(why)); };
+
+  private:
     ScheduledRecording();
     ~ScheduledRecording();
 
-    static void signalChange(int recordid);
-    // Use -1 for recordid when all recordids are potentially
-    // affected, such as when the program table is updated.
-    // Use 0 for recordid when a reschdule isn't specific to a single
-    // recordid, such as when a recording type priority is changed.
+    static void SendReschedule(const QStringList &request);
+    static QStringList BuildMatchRequest(uint recordid, uint sourceid, 
+              uint mplexid, const QDateTime &maxstarttime, const QString &why);
+    static QStringList BuildCheckRequest(const RecordingInfo &recinfo,
+                                         const QString &why);
+    static QStringList BuildPlaceRequest(const QString &why);
 };
 
 #endif

mythtv/libs/libmythtv/tv_rec.cpp

@@ -2668,7 +2668,7 @@ void TVRec::NotifySchedulerOfRecording(RecordingInfo *rec)
     rec->AddHistory(false);
 
     // + save RecordingRule so that we get a recordid
-    //   (don't allow signalChange(), avoiding unneeded reschedule)
+    //   (don't allow RescheduleMatch(), avoiding unneeded reschedule)
     rec->GetRecordingRule()->Save(false);
 
     // + save recordid to recorded entry

mythtv/programs/mythbackend/main_helpers.cpp

@@ -370,7 +370,10 @@ int handle_command(const MythBackendCommandLineParser &cmdline)
         }
 
         verboseMask |= VB_SCHEDULE;
+        LogLevel_t oldLogLevel = logLevel;
+        logLevel = LOG_DEBUG;
         sched->PrintList(true);
+        logLevel = oldLogLevel;
         delete sched;
         return GENERIC_EXIT_OK;
     }
@@ -381,7 +384,8 @@ int handle_command(const MythBackendCommandLineParser &cmdline)
         if (gCoreContext->ConnectToMasterServer())
         {
             LOG(VB_GENERAL, LOG_INFO, "Connected to master for reschedule");
-            ScheduledRecording::signalChange(-1);
+            ScheduledRecording::RescheduleMatch(0, 0, 0, QDateTime(),
+                                                "MythBackendCommand");
             ok = true;
         }
         else