updated Copy and Migration jobs

Issues #220: Copy job definition requires unnecessary settings

manuals/en/main/director-resource-job-definitions.tex

@@ -113,7 +113,7 @@
 }
 
 \defDirective{Dir}{Job}{Cancel Lower Level Duplicates}{}{}{%
-If \linkResourceDirective{Dir}{Job}{Allow Duplicates Jobs} is set to \parameter{no} and this
+If \linkResourceDirective{Dir}{Job}{Allow Duplicate Jobs} is set to \parameter{no} and this
 directive is set to \parameter{yes}, Bareos will choose between duplicated
 jobs the one with the highest level.  For example, it will cancel a
 previous Incremental to run a Full backup.  It works only for Backup
@@ -125,13 +125,13 @@
 }
 
 \defDirective{Dir}{Job}{Cancel Queued Duplicates}{}{}{%
-If \linkResourceDirective{Dir}{Job}{Allow Duplicates Jobs} is set to \parameter{no} and
+If \linkResourceDirective{Dir}{Job}{Allow Duplicate Jobs} is set to \parameter{no} and
 if this directive is set to \parameter{yes} any job that is
 already queued to run but not yet running will be canceled.
 }
 
 \defDirective{Dir}{Job}{Cancel Running Duplicates}{}{}{%
-If \linkResourceDirective{Dir}{Job}{Allow Duplicates Jobs} is set to \parameter{no} and
+If \linkResourceDirective{Dir}{Job}{Allow Duplicate Jobs} is set to \parameter{no} and
 if this directive is set to \parameter{yes} any job that is already running
 will be canceled.
 }
@@ -621,6 +621,16 @@
 different Clients or different Jobs, you will probably want to use
 Pools.  For additional details, see the \nameref{DirectorResourcePool}
 of this chapter.  This directive is required.
+
+In case of a Copy or Migration job,
+   this setting determines what Pool will be examined
+   for finding JobIds to migrate.  The exception to this is when
+   \linkResourceDirective{Dir}{Job}{Selection Type} = SQLQuery, 
+   and although a Pool directive must still be
+   specified, no Pool is used, unless you specifically include it in the
+   SQL query.  Note, in any case, the Pool resource defined by the Pool
+   directive must contain a \linkResourceDirective{Dir}{Pool}{Next Pool} = ... directive to define the
+   Pool to which the data will be migrated.
 }
 
 \defDirective{Dir}{Job}{Prefer Mounted Volumes}{}{}{%
@@ -748,7 +758,9 @@
 Pool resource.
 }
 
-\defDirective{Dir}{Job}{Purge Migration Job}{}{}{
+\defDirective{Dir}{Job}{Purge Migration Job}{}{}{%
+  This directive may be added to the Migration Job definition in the Director
+  configuration file to purge the job migrated at the end of a migration.
 }
 
 \defDirective{Dir}{Job}{Regex Where}{}{}{%
@@ -1302,10 +1314,109 @@
 additional details, see \nameref{DirectorResourceSchedule}.
 }
 
-\defDirective{Dir}{Job}{Selection Pattern}{}{}{
-}
+\defDirective{Dir}{Job}{Selection Pattern}{}{}{%
+Selection Patterns is only used for Copy and Migration jobs, see \nameref{MigrationChapter}.
+The interpretation of its value depends on the selected \linkResourceDirective{Dir}{Job}{Selection Type}.
+
+  For the OldestVolume and SmallestVolume, this
+  Selection pattern is not used (ignored).
+
+  For the Client, Volume, and Job
+  keywords, this pattern must be a valid regular expression that will filter
+  the appropriate item names found in the Pool.
+
+  For the SQLQuery keyword, this pattern must be a valid \command{SELECT} SQL statement
+  that returns JobIds.
+}
+
+\defDirective{Dir}{Job}{Selection Type}{}{}{%
+Selection Type is only used for Copy and Migration jobs, see \nameref{MigrationChapter}.
+It determines how a migration job
+  will go about selecting what JobIds to migrate. In most cases, it is
+  used in conjunction with a \linkResourceDirective{Dir}{Job}{Selection Pattern}
+  to give you fine
+  control over exactly what JobIds are selected.
+  The possible values are:
+  \begin{description}
+  \item [SmallestVolume] This selection keyword selects the volume with the
+        fewest bytes from the Pool to be migrated.  The Pool to be migrated
+        is the Pool defined in the Migration Job resource.  The migration
+        control job will then start and run one migration backup job for
+        each of the Jobs found on this Volume.  The Selection Pattern, if
+        specified, is not used.
+
+  \item [OldestVolume] This selection keyword selects the volume with the
+        oldest last write time in the Pool to be migrated.  The Pool to be
+        migrated is the Pool defined in the Migration Job resource.  The
+        migration control job will then start and run one migration backup
+        job for each of the Jobs found on this Volume.  The Selection
+        Pattern, if specified, is not used.
+
+  \item [Client] The Client selection type, first selects all the Clients
+        that have been backed up in the Pool specified by the Migration
+        Job resource, then it applies the \linkResourceDirective{Dir}{Job}{Selection Pattern}
+        as a regular expression to the list of Client names, giving
+        a filtered Client name list.  All jobs that were backed up for those
+        filtered (regexed) Clients will be migrated.
+        The migration control job will then start and run one migration
+        backup job for each of the JobIds found for those filtered Clients.
+
+  \item [Volume] The Volume selection type, first selects all the Volumes
+        that have been backed up in the Pool specified by the Migration
+        Job resource, then it applies the \linkResourceDirective{Dir}{Job}{Selection Pattern}
+        as a regular expression to the list of Volume names, giving
+        a filtered Volume list.  All JobIds that were backed up for those
+        filtered (regexed) Volumes will be migrated.
+        The migration control job will then start and run one migration
+        backup job for each of the JobIds found on those filtered Volumes.
+
+  \item [Job] The Job selection type, first selects all the Jobs (as
+        defined on the \linkResourceDirective{Dir}{Job}{Name} directive in a Job resource)
+        that have been backed up in the Pool specified by the Migration
+        Job resource, then it applies the \linkResourceDirective{Dir}{Job}{Selection Pattern}
+        as a regular expression to the list of Job names, giving
+        a filtered Job name list.  All JobIds that were run for those
+        filtered (regexed) Job names will be migrated.  Note, for a given
+        Job named, they can be many jobs (JobIds) that ran.
+        The migration control job will then start and run one migration
+        backup job for each of the Jobs found.
+
+  \item [SQLQuery] The SQLQuery selection type, used the 
+        \linkResourceDirective{Dir}{Job}{Selection Pattern}
+        as an SQL query to obtain the JobIds to be migrated.
+        The Selection Pattern must be a valid SELECT SQL statement for your
+        SQL engine, and it must return the JobId as the first field
+        of the SELECT.
+
+  \item [PoolOccupancy] This selection type will cause the Migration job
+        to compute the total size of the specified pool for all Media Types
+        combined. If it exceeds the \linkResourceDirective{Dir}{Pool}{Migration High Bytes} 
+        defined in
+        the Pool, the Migration job will migrate all JobIds beginning with
+        the oldest Volume in the pool (determined by Last Write time) until
+        the Pool bytes drop below the \linkResourceDirective{Dir}{Pool}{Migration Low Bytes} 
+        defined in the
+        Pool. This calculation should be consider rather approximative because
+        it is made once by the Migration job before migration is begun, and
+        thus does not take into account additional data written into the Pool
+        during the migration.  In addition, the calculation of the total Pool
+        byte size is based on the Volume bytes saved in the Volume (Media) database
+        entries. The bytes calculate for Migration is based on the value stored
+        in the Job records of the Jobs to be migrated. These do not include the
+        Storage daemon overhead as is in the total Pool size. As a consequence,
+        normally, the migration will migrate more bytes than strictly necessary.
+
+  \item [PoolTime] The PoolTime selection type will cause the Migration job to
+        look at the time each JobId has been in the Pool since the job ended.
+        All Jobs in the Pool longer than the time specified on 
+        \linkResourceDirective{Dir}{Pool}{Migration Time}
+        directive in the Pool resource will be migrated.
+
+  \item [PoolUncopiedJobs] This selection which copies all jobs from a pool
+        to an other pool which were not copied before is available only for copy Jobs.
+
+  \end{description}
 
-\defDirective{Dir}{Job}{Selection Type}{}{}{
 }
 
 \defDirective{Dir}{Job}{Spool Attributes}{}{}{%
@@ -1421,6 +1532,30 @@
 Run an admin Job. An {\bf Admin} job can  be used to periodically run catalog
 pruning, if you  do not want to do it at the end of each {\bf Backup}  Job.
 Although an Admin job is recorded in the  catalog, very little data is saved.
+
+\item [Migrate]
+   defines the job that is run as being a
+   Migration Job.  A Migration Job is a sort of control job and does not have
+   any Files associated with it, and in that sense they are more or less like
+   an Admin job.  Migration jobs simply check to see if there is anything to
+   Migrate then possibly start and control new Backup jobs to migrate the data
+   from the specified Pool to another Pool.  Note, any original JobId that
+   is migrated will be marked as having been migrated, and the original
+   JobId can nolonger be used for restores; all restores will be done from
+   the new migrated Job.
+
+\item [Copy]
+   defines the job that is run as being a
+   Copy Job.  A Copy Job is a sort of control job and does not have
+   any Files associated with it, and in that sense they are more or less like
+   an Admin job.  Copy jobs simply check to see if there is anything to
+   Copy then possibly start and control new Backup jobs to copy the data
+   from the specified Pool to another Pool.  Note that when a copy is
+   made, the original JobIds are left unchanged. The new copies can not
+   be used for restoration unless you specifically choose them by JobId.
+   If you subsequently delete a JobId that has a copy, the copy will be
+   automatically upgraded to a Backup rather than a Copy, and it will
+   subsequently be used for restoration.
 \end{description}
 }
 

manuals/en/main/director-resource-pool-definitions.tex

@@ -56,9 +56,6 @@
 accordingly.
 }
 
-\defDirective{Dir}{Pool}{Copy Pool}{}{}{%
-}
-
 \defDirective{Dir}{Pool}{Description}{}{}{%
 }
 
@@ -219,12 +216,35 @@
 }
 
 \defDirective{Dir}{Pool}{Migration High Bytes}{}{}{%
+This directive specifies the number of bytes in the Pool which will
+   trigger a migration if \linkResourceDirective{Dir}{Job}{Selection Type} = PoolOccupancy
+   has been specified. The fact that the Pool
+   usage goes above this level does not automatically trigger a migration
+   job. However, if a migration job runs and has the PoolOccupancy selection
+   type set, the Migration High Bytes will be applied.  Bareos does not
+   currently restrict a pool to have only a single \linkResourceDirective{Dir}{Storage}{Media Type}, so you
+   must keep in mind that if you mix Media Types in a Pool, the results
+   may not be what you want, as the Pool count of all bytes will be
+   for all Media Types combined.
 }
 
 \defDirective{Dir}{Pool}{Migration Low Bytes}{}{}{%
+This directive specifies the number of bytes in the Pool which will
+   stop a migration if \linkResourceDirective{Dir}{Job}{Selection Type} = PoolOccupancy
+   has been specified and triggered by more than 
+   \linkResourceDirective{Dir}{Pool}{Migration High Bytes}
+   being in the pool. In other words, once a migration job
+   is started with {\bf PoolOccupancy} migration selection and it
+   determines that there are more than Migration High Bytes, the
+   migration job will continue to run jobs until the number of
+   bytes in the Pool drop to or below Migration Low Bytes.
 }
 
 \defDirective{Dir}{Pool}{Migration Time}{}{}{%
+If \linkResourceDirective{Dir}{Job}{Selection Type} = PoolTime, 
+the time specified here will be used. 
+If the previous Backup Job or Jobs selected have been in the Pool longer than
+the specified time, then they will be migrated.
 }
 
 \defDirective{Dir}{Pool}{Minimum Blocksize}{}{}{%
@@ -237,8 +257,11 @@
 }
 
 \defDirective{Dir}{Pool}{Next Pool}{}{}{%
-This directive specifies the Next pool a Migration or Copy Job
+This directive specifies the pool a Migration or Copy Job
 and a Virtual Backup Job will write their data too.
+This directive is required to define the Pool into which
+   the data will be migrated. Without this directive, the migration job
+   will terminate in error.
 }
 
 \defDirective{Dir}{Pool}{Pool Type}{}{}{%
@@ -386,6 +409,9 @@
 in the Job. This Storage resource definition is not required by either
 the Job resource or in the Pool, but it must be specified in
 one or the other.  If not configuration error will result.
+  We highly recommend
+   that you define the Storage resource to be used in the Pool rather
+   than elsewhere (job, schedule run, ...).
 }
 
 \defDirective{Dir}{Pool}{Use Catalog}{}{}{%