From 9ada8f1f99d2453bb53467094096247941c4316d Mon Sep 17 00:00:00 2001 From: Frank Ueberschar Date: Thu, 25 Jul 2019 12:59:15 +0200 Subject: [PATCH] docs: corrected database table rows in chapter catalog --- .../manuals/source/DeveloperGuide/catalog.rst | 860 ++++++++---------- .../catalog/schema_bareos_postgresql.puml | 1 - 2 files changed, 376 insertions(+), 485 deletions(-) diff --git a/docs/manuals/source/DeveloperGuide/catalog.rst b/docs/manuals/source/DeveloperGuide/catalog.rst index 3452e40843e..abb467378fc 100644 --- a/docs/manuals/source/DeveloperGuide/catalog.rst +++ b/docs/manuals/source/DeveloperGuide/catalog.rst @@ -1,57 +1,41 @@ -Catalog Services +catalog Services ================ General ------- -This chapter is intended to be a technical discussion of the Catalog +This chapter is intended to be a technical discussion of the catalog services and as such is not targeted at end users but rather at -developers and system administrators that want or need to know more of +developers and system administrators who want or need to know more of the working details of **Bareos**. -The **Bareos Catalog** services consist of the programs that provide the -SQL database engine for storage and retrieval of all information -concerning files that were backed up and their locations on the storage -media. - -We have investigated the possibility of using the following SQL engines -for Bareos: Beagle, mSQL, GNU SQL, PostgreSQL, SQLite, Oracle, and -MySQL. Each presents certain problems with either licensing or maturity. -At present, we have chosen for development purposes to use MySQL, -PostgreSQL and SQLite. MySQL was chosen because it is fast, proven to be -reliable, widely used, and actively being developed. MySQL is released -under the GNU GPL license. PostgreSQL was chosen because it is a -full-featured, very mature database, and because Dan Langille did the -Bareos driver for it. PostgreSQL is distributed under the BSD license. +Currently, we prefer the usage of PostgreSQL. Therefore our support +for SQLite or other databases could be discontinued in the future. +PostgreSQL was chosen because it is a full-featured, very mature database, +and because Dan Langille did the Bareos driver for it. + SQLite was chosen because it is small, efficient, and can be directly embedded in **Bareos** thus requiring much less effort from the system administrator or person building **Bareos**. In our testing SQLite has performed very well, and for the functions that we use, it has never encountered any errors except that it does not appear to handle databases larger than 2GBytes. That said, we would not recommend it for -serious production use. - -The Bareos SQL code has been written in a manner that will allow it to -be easily modified to support any of the current SQL database systems on -the market (for example: mSQL, iODBC, unixODBC, Solid, OpenLink ODBC, -EasySoft ODBC, InterBase, Oracle8, Oracle7, and DB2). +serious production use. Nonetheless SQLite is very suitable for test +environments. -If you do not specify either **``--``\ with-mysql** or -**``--``\ with-postgresql** or **``--``\ with-sqlite** on the -./configure line, Bareos will use its minimalist internal database. This -database is kept for build reasons but is no longer supported. Bareos -**requires** one of the three databases (MySQL, PostgreSQL, or SQLite) -to run. +Bareos **requires** one of the three databases (MySQL, PostgreSQL, or SQLite) +to run. Therefore it is mandatory to specify one of them for the cmake +configuration step, i.e.: ``-Dpostgresql=yes``. Filenames and Maximum Filename Length ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In general, either MySQL, PostgreSQL or SQLite permit storing arbitrary long path names and file names in the catalog database. In practice, -there still may be one or two places in the Catalog interface code that +there still may be one or two places in the catalog interface code that restrict the maximum path length to 512 characters and the maximum file name length to 512 characters. These restrictions are believed to have -been removed. Please note, these restrictions apply only to the Catalog +been removed. Please note, these restrictions apply only to the catalog database and thus to your ability to list online the files saved during any job. All information received and stored by the Storage daemon (normally on tape) allows and handles arbitrarily long path and @@ -60,174 +44,93 @@ filenames. Database Table Design ~~~~~~~~~~~~~~~~~~~~~ -All discussions that follow pertain to the MySQL database. The details -for the PostgreSQL and SQLite databases are essentially identical except -for that all fields in the SQLite database are stored as ASCII text and -some of the database creation statements are a bit different. The -details of the internal Bareos catalog are not discussed here. +All discussions that follow pertain to the PostgreSQL database. -Because the Catalog database may contain very large amounts of data for +Because the catalog database may contain very large amounts of data for large sites, we have made a modest attempt to normalize the data tables to reduce redundant information. While reducing the size of the database significantly, it does, unfortunately, add some complications to the structures. -In simple terms, the Catalog database must contain a record of all Jobs +In simple terms, the catalog database must contain a record of all Jobs run by Bareos, and for each Job, it must maintain a list of all files saved, with their File Attributes (permissions, create date, …), and the location and Media on which the file is stored. This is seemingly a -simple task, but it represents a huge amount interlinked data. Note: the -list of files and their attributes is not maintained when using the -internal Bareos database. The data stored in the File records, which -allows the user or administrator to obtain a list of all files backed up -during a job, is by far the largest volume of information put into the -Catalog database. - -Although the Catalog database has been designed to handle backup data +simple task, but it represents a huge amount of interlinked data. +The data stored in the File records, which allows the user or +administrator to obtain a list of all files backed up during a job, +is by far the largest volume of information put into the catalog database. + +Although the catalog database has been designed to handle backup data for multiple clients, some users may want to maintain multiple databases, one for each machine to be backed up. This reduces the risk of confusion of accidental restoring a file to the wrong machine as well as reducing the amount of data in a single database, thus increasing efficiency and reducing the impact of a lost or damaged database. -Sequence of Creation of Records for a Save Job ----------------------------------------------- - -Start with StartDate, ClientName, Filename, Path, Attributes, MediaName, -MediaCoordinates. (PartNumber, NumParts). In the steps below, “Create -new” means to create a new record whether or not it is unique. “Create -unique” means each record in the database should be unique. Thus, one -must first search to see if the record exists, and only if not should a -new one be created, otherwise the existing RecordId should be used. - -1. Create new Job record with StartDate; save JobId - -2. Create unique Media record; save MediaId - -3. Create unique Client record; save ClientId - -4. Create unique Filename record; save FilenameId - -5. Create unique Path record; save PathId - -6. Create unique Attribute record; save AttributeId store ClientId, - FilenameId, PathId, and Attributes - -7. Create new File record store JobId, AttributeId, MediaCoordinates, - etc - -8. Repeat steps 4 through 8 for each file - -9. Create a JobMedia record; save MediaId - -10. Update Job record filling in EndDate and other Job statistics - Database Tables --------------- -Filename -~~~~~~~~ - -+-------------+-----------+-------------+ -| Column Name | Data Type | Remark | -+=============+===========+=============+ -| FilenameId | integer | Primary Key | -+-------------+-----------+-------------+ -| Name | Blob | Filename | -+-------------+-----------+-------------+ - -The **Filename** table shown above contains the name of each file backed -up with the path removed. If different directories or machines contain -the same filename, only one copy will be saved in this table. - Path ~~~~ +The **Path** table contains shown above the path or directory names of all +directories on the system or systems. + +-------------+-----------+-------------+ | Column Name | Data Type | Remark | +=============+===========+=============+ -| PathId | integer | Primary Key | +| PathId | serial | Primary Key | +-------------+-----------+-------------+ -| Path | Blob | Full Path | +| Path | text | Full Path | +-------------+-----------+-------------+ -The **Path** table contains shown above the path or directory names of -all directories on the system or systems. The filename and any MSDOS -disk name are stripped off. As with the filename, only one copy of each -directory name is kept regardless of how many machines or drives have -the same directory. These path names should be stored in Unix path name -format. - -Some simple testing on a Linux file system indicates that separating the -filename and the path may be more complication than is warranted by the -space savings. For example, this system has a total of 89,097 files, -60,467 of which have unique filenames, and there are 4,374 unique paths. - -Finding all those files and doing two stats() per file takes an average -wall clock time of 1 min 35 seconds on a 400MHz machine running RedHat -6.1 Linux. - -Finding all those files and putting them directly into a MySQL database -with the path and filename defined as TEXT, which is variable length up -to 65,535 characters takes 19 mins 31 seconds and creates a 27.6 MByte -database. - -Doing the same thing, but inserting them into Blob fields with the -filename indexed on the first 30 characters and the path name indexed on -the 255 (max) characters takes 5 mins 18 seconds and creates a 5.24 MB -database. Rerunning the job (with the database already created) takes -about 2 mins 50 seconds. - -Running the same as the last one (Path and Filename Blob), but Filename -indexed on the first 30 characters and the Path on the first 50 -characters (linear search done there after) takes 5 mins on the average -and creates a 3.4 MB database. Rerunning with the data already in the DB -takes 3 mins 35 seconds. - -Finally, saving only the full path name rather than splitting the path -and the file, and indexing it on the first 50 characters takes 6 mins 43 -seconds and creates a 7.35 MB database. +The filename and any disk name are stripped off. As with the filename, +only one copy of each directory name is kept regardless of how many machines +or drives have the same directory. These path names should be stored in Unix +path name format. File ~~~~ -+-------------+-----------+---------------------------------------+ -| Column Name | Data Type | Remark | -+=============+===========+=======================================+ -| FileId | integer | Primary Key | -+-------------+-----------+---------------------------------------+ -| FileIndex | integer | The sequential file number in the Job | -+-------------+-----------+---------------------------------------+ -| JobId | integer | Link to Job Record | -+-------------+-----------+---------------------------------------+ -| PathId | integer | Link to Path Record | -+-------------+-----------+---------------------------------------+ -| FilenameId | integer | Link to Filename Record | -+-------------+-----------+---------------------------------------+ -| DeltaSeq | smallint | | -+-------------+-----------+---------------------------------------+ -| MarkId | integer | Used to mark files during Verify Jobs | -+-------------+-----------+---------------------------------------+ -| LStat | tinyblob | File attributes in base64 encoding | -+-------------+-----------+---------------------------------------+ -| MD5 | tinyblob | MD5/SHA1 signature in base64 encoding | -+-------------+-----------+---------------------------------------+ - -The **File** table shown above contains one entry for each file backed -up by Bareos. Thus a file that is backed up multiple times (as is -normal) will have multiple entries in the File table. This will probably +The **File** table contains one entry for each file backed up by Bareos. + ++-------------+---------------+---------------------------------------+ +| Column Name | Data Type | Remark | ++=============+===============+=======================================+ +| FileId | serial | Primary Key | ++-------------+---------------+---------------------------------------+ +| FileIndex | integer | The sequential file number in the Job | ++-------------+---------------+---------------------------------------+ +| JobId | integer | Link to Job Record | ++-------------+---------------+---------------------------------------+ +| PathId | integer | Link to Path Record | ++-------------+---------------+---------------------------------------+ +| DeltaSeq | smallint | | ++-------------+---------------+---------------------------------------+ +| MarkId | integer | Used to mark files during Verify Jobs | ++-------------+---------------+---------------------------------------+ +| Fhinofo | numeric(20) | | ++-------------+---------------+---------------------------------------+ +| Fhnode | numeric(20) | | ++-------------+---------------+---------------------------------------+ +| LStat | text | File attributes in base64 encoding | ++-------------+---------------+---------------------------------------+ +| MD5 | text | MD5/SHA1 signature in base64 encoding | ++-------------+---------------+---------------------------------------+ +| name | text | | ++-------------+---------------+---------------------------------------+ + +File that is backed up multiple times (as is normal) will have multiple +entries in the File table. This will probably be the table with the most number of records. Consequently, it is essential to keep the size of this record to an absolute minimum. At the same time, this table must contain all the information (or pointers to -the information) about the file and where it is backed up. Since a file -may be backed up many times without having changed, the path and -filename are stored in separate tables. +the information) about the file and where it is backed up. This table contains by far the largest amount of information in the -Catalog database, both from the stand point of number of records, and -the stand point of total database size. As a consequence, the user must -take care to periodically reduce the number of File records using the -**retention** command in the Console program. +catalog database, both from the stand point of number of records, and +the stand point of total database size. As MD5 hash (also termed message digests) consists of 128-bit (16-byte). A typical format (eg. ``md5sum``, …) to represent them is as a sequence @@ -240,96 +143,83 @@ To compare a Bareos digest with command line tools, you have to use openssl dgst -md5 -binary $PATH_OF_YOUR_FILE | base64 -Also you have to note, that even the table column is named **MD5**, it -used to store any kind of digest (MD5, SHA1, …). This is not directly -indicated by the value, however, you can determine the type depending of -the length of the digest. +Please note, even the table column is named **MD5**, it is used to store +any kind of digest (MD5, SHA1, …). This is not directly indicated by the +value, however, you can determine the type depending of the length of +the digest. -Job / JobHisto -~~~~~~~~~~~~~~ +Job +~~~ -+----------+----------------+------------------------------------------+ -| Column | Data Type | Remark | -| Name | | | -+==========+================+==========================================+ -| JobId | integer | Primary Key | -+----------+----------------+------------------------------------------+ -| Job | tinyblob | Unique Job Name | -+----------+----------------+------------------------------------------+ -| Name | tinyblob | Job Name | -+----------+----------------+------------------------------------------+ -| Type | binary(1) | Job Type: Backup, Copy, Clone, Archive, | -| | | Migration | -+----------+----------------+------------------------------------------+ -| Level | binary(1) | Job Level | -+----------+----------------+------------------------------------------+ -| ClientId | integer | Client index | -+----------+----------------+------------------------------------------+ -| JobStatu | binary(1) | Job Termination Status | -| s | | | -+----------+----------------+------------------------------------------+ -| SchedTim | datetime | Time/date when Job scheduled | -| e | | | -+----------+----------------+------------------------------------------+ -| StartTim | datetime | Time/date when Job started | -| e | | | -+----------+----------------+------------------------------------------+ -| EndTime | datetime | Time/date when Job ended | -+----------+----------------+------------------------------------------+ -| ReadEndT | datetime | Time/date when original Job ended | -| ime | | | -+----------+----------------+------------------------------------------+ -| JobTDate | bigint | Start day in Unix format but 64 bits; | -| | | used for Retention period. | -+----------+----------------+------------------------------------------+ -| VolSessi | integer | Unique Volume Session ID | -| onId | | | -+----------+----------------+------------------------------------------+ -| VolSessi | integer | Unique Volume Session Time | -| onTime | | | -+----------+----------------+------------------------------------------+ -| JobFiles | integer | Number of files saved in Job | -+----------+----------------+------------------------------------------+ -| JobBytes | bigint | Number of bytes saved in Job | -+----------+----------------+------------------------------------------+ -| JobError | integer | Number of errors during Job | -| s | | | -+----------+----------------+------------------------------------------+ -| JobMissi | integer | Number of files not saved (not yet used) | -| ngFiles | | | -+----------+----------------+------------------------------------------+ -| PoolId | integer | Link to Pool Record | -+----------+----------------+------------------------------------------+ -| FileSetI | integer | Link to FileSet Record | -| d | | | -+----------+----------------+------------------------------------------+ -| PrioJobI | integer | Link to prior Job Record when migrated | -| d | | | -+----------+----------------+------------------------------------------+ -| PurgedFi | tiny integer | Set when all File records purged | -| les | | | -+----------+----------------+------------------------------------------+ -| HasBase | tiny integer | Set when Base Job run | -+----------+----------------+------------------------------------------+ - -The **Job** table contains one record for each Job run by Bareos. Thus -normally, there will be one per day per machine added to the database. -Note, the JobId is used to index Job records in the database, and it -often is shown to the user in the Console program. However, care must be -taken with its use as it is not unique from database to database. For -example, the user may have a database for Client data saved on machine -Rufus and another database for Client data saved on machine Roxie. In -this case, the two database will each have JobIds that match those in -another database. For a unique reference to a Job, see Job below. +The **Job** table contains one record for each Job run by Bareos. + ++----------------+----------------+------------------------------------------+ +| Column | Data Type | Remark | +| Name | | | ++================+================+==========================================+ +| JobId | serial | Primary Key | ++----------------+----------------+------------------------------------------+ +| Job | text | Unique Job Name | ++----------------+----------------+------------------------------------------+ +| Name | text | Job Name | ++----------------+----------------+------------------------------------------+ +| Type | char(1) | Job Type: Backup, Copy, Clone, Archive, | +| | | Migration | ++----------------+----------------+------------------------------------------+ +| Level | char(1) | Job Level | ++----------------+----------------+------------------------------------------+ +| ClientId | integer | Client index | ++----------------+----------------+------------------------------------------+ +| JobStatus | char(1) | Job Termination Status | ++----------------+----------------+------------------------------------------+ +| SchedTime | timestamp | Time/date when Job scheduled | ++----------------+----------------+------------------------------------------+ +| StartTime | timestamp | Time/date when Job started | ++----------------+----------------+------------------------------------------+ +| EndTime | timestamp | Time/date when Job ended | ++----------------+----------------+------------------------------------------+ +| RealEndTime | timestamp | Time/date when original Job ended | ++----------------+----------------+------------------------------------------+ +| JobTDate | bigint | Start day in Unix format but 64 bits; | +| | | used for Retention period. | ++----------------+----------------+------------------------------------------+ +| VolSessionId | integer | Unique Volume Session ID | ++----------------+----------------+------------------------------------------+ +| VolSessionTime | integer | Unique Volume Session Time | ++----------------+----------------+------------------------------------------+ +| JobFiles | integer | Number of files saved in Job | ++----------------+----------------+------------------------------------------+ +| JobBytes | bigint | Number of bytes saved in Job | ++----------------+----------------+------------------------------------------+ +| ReadBytes | bigint | | ++----------------+----------------+------------------------------------------+ +| JobErrors | integer | Number of errors during Job | ++----------------+----------------+------------------------------------------+ +| JobMissinFiles | integer | Number of files not saved (not yet used) | ++----------------+----------------+------------------------------------------+ +| PoolId | integer | Link to Pool Record | ++----------------+----------------+------------------------------------------+ +| FileSetId | integer | Link to FileSet Record | ++----------------+----------------+------------------------------------------+ +| PriorJobId | integer | Link to prior Job Record when migrated | ++----------------+----------------+------------------------------------------+ +| PurgedFiles | smallint | Set when all File records purged | ++----------------+----------------+------------------------------------------+ +| HasBase | smallint | Set when Base Job run | ++----------------+----------------+------------------------------------------+ +| HasCache | smallint | | ++----------------+----------------+------------------------------------------+ +| Reviewed | smallint | | ++----------------+----------------+------------------------------------------+ +| Comment | text | | ++----------------+----------------+------------------------------------------+ The Name field of the Job record corresponds to the Name resource record -given in the Director’s configuration file. Thus it is a generic name, -and it will be normal to find many Jobs (or even all Jobs) with the same -Name. +given in the Director’s configuration file. The Job field contains a combination of the Name and the schedule time of the Job by the Director. Thus for a given Director, even with -multiple Catalog databases, the Job will contain a unique name that +multiple catalog databases, the Job will contain a unique name that represents the Job. For a given Storage daemon, the VolSessionId and VolSessionTime form a @@ -338,25 +228,25 @@ Directors are using the same Storage daemon. The JobStatus field specifies how the job terminated. -The JobHisto table is the same as the Job table, but it keeps long term -statistics (i.e. it is not pruned with the Job). - FileSet ~~~~~~~ +The **FileSet** table contains one entry for each FileSet that is used. + +-------------+-----------+-------------------------------+ | Column Name | Data Type | Remark | +=============+===========+===============================+ -| FileSetId | integer | Primary Key | +| FileSetId | serial | Primary Key | ++-------------+-----------+-------------------------------+ +| FileSet | text | FileSet name | +-------------+-----------+-------------------------------+ -| FileSet | tinyblob | FileSet name | +| FileSetText | text | | +-------------+-----------+-------------------------------+ -| MD5 | tinyblob | MD5 checksum of FileSet | +| MD5 | text | MD5 checksum of FileSet | +-------------+-----------+-------------------------------+ -| CreateTime | datetime | Time and date Fileset created | +| CreateTime | timestamp | Time and date Fileset created | +-------------+-----------+-------------------------------+ -The **FileSet** table contains one entry for each FileSet that is used. The MD5 signature is kept to ensure that if the user changes anything inside the FileSet, it will be detected and the new FileSet will be used. This is particularly important when doing an incremental update. @@ -366,57 +256,51 @@ backup is done prior to the next incremental. JobMedia ~~~~~~~~ -+--------+--------+---------------------------------------------------+ -| Column | Data | Remark | -| Name | Type | | -+========+========+===================================================+ -| JobMed | intege | Primary Key | -| iaId | r | | -+--------+--------+---------------------------------------------------+ -| JobId | intege | Link to Job Record | -| | r | | -+--------+--------+---------------------------------------------------+ -| MediaI | intege | Link to Media Record | -| d | r | | -+--------+--------+---------------------------------------------------+ -| FirstI | intege | The index (sequence number) of the first file | -| ndex | r | written for this Job to the Media | -+--------+--------+---------------------------------------------------+ -| LastIn | intege | The index of the last file written for this Job | -| dex | r | to the Media | -+--------+--------+---------------------------------------------------+ -| StartF | intege | Tape: the physical media file mark number of the | -| ile | r | first block written for this Job. | -+--------+--------+---------------------------------------------------+ -| | | Other: upper 32-bit of the position of the first | -| | | block written for this Job. | -+--------+--------+---------------------------------------------------+ -| EndFil | intege | Tape: the physical media file mark number of the | -| e | r | last block written for this Job. | -+--------+--------+---------------------------------------------------+ -| | | Other: upper 32-bit of the position of the last | -| | | block written for this Job. | -+--------+--------+---------------------------------------------------+ -| StartB | intege | Tape: the number of the first block written for | -| lock | r | this Job | -+--------+--------+---------------------------------------------------+ -| | | Other: lower 32-bit of the position of the first | -| | | block written for this Job. | -+--------+--------+---------------------------------------------------+ -| Endblo | intege | Tape: the number of the last block written for | -| ck | r | this Job | -+--------+--------+---------------------------------------------------+ -| | | Other: lower 32-bit of the position of the last | -| | | block written for this Job. | -+--------+--------+---------------------------------------------------+ -| VolInd | intege | The Volume use sequence number within the Job | -| ex | r | | -+--------+--------+---------------------------------------------------+ - The **JobMedia** table contains one entry at the following: start of the job, start of each new tape file mark, start of each new tape, end of the job. You will have 2 or more JobMedia records per Job. ++-------------+-------------+---------------------------------------------------+ +| Column | Data | Remark | +| Name | Type | | ++=============+=============+===================================================+ +| JobMediaId | integer | Primary Key | ++-------------+-------------+---------------------------------------------------+ +| JobId | integer | Link to Job Record | ++-------------+-------------+---------------------------------------------------+ +| MediaId | integer | Link to Media Record | ++-------------+-------------+---------------------------------------------------+ +| FirstIndex | integer | The index (sequence number) of the first file | +| | | written for this Job to the Media | ++-------------+-------------+---------------------------------------------------+ +| LastIndex | integer | The index of the last file written for this Job | +| | | to the Media | ++-------------+-------------+---------------------------------------------------+ +| StartFile | integer | | *Tape*: The physical media file mark number of | +| | | the first block written for this Job. | +| | | | *Other*: Upper 32-bit of the position of the | +| | | first block written for this Job. | ++-------------+-------------+---------------------------------------------------+ +| EndFile | integer | | *Tape*: The physical media file mark number of | +| | | the last block written for this Job | +| | | | *Other*: Upper 32-bit of the position of the | +| | | last block written for this Job | ++-------------+-------------+---------------------------------------------------+ +| StartBlock | integer | | *Tape*: The number of the first block written | +| | | for this Job | +| | | | *Other*: Lower 32-bit of the position of the | +| | | first block written for this Job | ++-------------+-------------+---------------------------------------------------+ +| Endblock | integer | | *Tape*: The number of the last block written for| +| | | this Job | +| | | | *Other*: Lower 32-bit of the position of the | +| | | last block written for this Job | ++-------------+-------------+---------------------------------------------------+ +| JobBytes | numeric(20) | The Volume use sequence number within the Job | ++-------------+-------------+---------------------------------------------------+ +| VolIndex | integer | The Volume use sequence number within the Job | ++-------------+-------------+---------------------------------------------------+ + Tape Volume ^^^^^^^^^^^ @@ -455,129 +339,134 @@ can also be stored between Start- and EndPosition. EndPosition - StartPosition >= JobSizeOnThisMedia -Volume (Media) +Media (Volume) ~~~~~~~~~~~~~~ -+-----------------+----------+-----------------------------------------+ -| Column Name | Data | Remark | -| | Type | | -+=================+==========+=========================================+ -| MediaId | integer | Primary Key | -+-----------------+----------+-----------------------------------------+ -| VolumeName | tinyblob | Volume name | -+-----------------+----------+-----------------------------------------+ -| Slot | integer | Autochanger Slot number or zero | -+-----------------+----------+-----------------------------------------+ -| PoolId | integer | Link to Pool Record | -+-----------------+----------+-----------------------------------------+ -| MediaType | tinyblob | The MediaType supplied by the user | -+-----------------+----------+-----------------------------------------+ -| MediaTypeId | integer | The MediaTypeId | -+-----------------+----------+-----------------------------------------+ -| LabelType | tinyint | The type of label on the Volume | -+-----------------+----------+-----------------------------------------+ -| FirstWritten | datetime | Time/date when first written | -+-----------------+----------+-----------------------------------------+ -| LastWritten | datetime | Time/date when last written | -+-----------------+----------+-----------------------------------------+ -| LabelDate | datetime | Time/date when tape labeled | -+-----------------+----------+-----------------------------------------+ -| VolJobs | integer | Number of jobs written to this media | -+-----------------+----------+-----------------------------------------+ -| VolFiles | integer | Number of files written to this media | -+-----------------+----------+-----------------------------------------+ -| VolBlocks | integer | Number of blocks written to this media | -+-----------------+----------+-----------------------------------------+ -| VolMounts | integer | Number of time media mounted | -+-----------------+----------+-----------------------------------------+ -| VolBytes | bigint | Number of bytes saved in Job | -+-----------------+----------+-----------------------------------------+ -| VolParts | integer | The number of parts for a Volume (DVD) | -+-----------------+----------+-----------------------------------------+ -| VolErrors | integer | Number of errors during Job | -+-----------------+----------+-----------------------------------------+ -| VolWrites | integer | Number of writes to media | -+-----------------+----------+-----------------------------------------+ -| MaxVolBytes | bigint | Maximum bytes to put on this media | -+-----------------+----------+-----------------------------------------+ -| VolCapacityByte | bigint | Capacity estimate for this volume | -| s | | | -+-----------------+----------+-----------------------------------------+ -| VolStatus | enum | Status of media: Full, Archive, Append, | -| | | Recycle, Read-Only, Disabled, Error, | -| | | Busy | -+-----------------+----------+-----------------------------------------+ -| Enabled | tinyint | Whether or not Volume can be written | -+-----------------+----------+-----------------------------------------+ -| Recycle | tinyint | Whether or not Bareos can recycle the | -| | | Volumes: Yes, No | -+-----------------+----------+-----------------------------------------+ -| ActionOnPurge | tinyint | What happens to a Volume after purging | -+-----------------+----------+-----------------------------------------+ -| VolRetention | bigint | 64 bit seconds until expiration | -+-----------------+----------+-----------------------------------------+ -| VolUseDureation | bigint | 64 bit seconds volume can be used | -+-----------------+----------+-----------------------------------------+ -| MaxVolJobs | integer | maximum jobs to put on Volume | -+-----------------+----------+-----------------------------------------+ -| MaxVolFiles | integer | maximume EOF marks to put on Volume | -+-----------------+----------+-----------------------------------------+ -| InChanger | tinyint | Whether or not Volume in autochanger | -+-----------------+----------+-----------------------------------------+ -| StorageId | integer | Storage record ID | -+-----------------+----------+-----------------------------------------+ -| DeviceId | integer | Device record ID | -+-----------------+----------+-----------------------------------------+ -| MediaAddressing | integer | Method of addressing media | -+-----------------+----------+-----------------------------------------+ -| VolReadTime | bigint | Time Reading Volume | -+-----------------+----------+-----------------------------------------+ -| VolWriteTime | bigint | Time Writing Volume | -+-----------------+----------+-----------------------------------------+ -| EndFile | integer | End File number of Volume | -+-----------------+----------+-----------------------------------------+ -| EndBlock | integer | End block number of Volume | -+-----------------+----------+-----------------------------------------+ -| LocationId | integer | Location record ID | -+-----------------+----------+-----------------------------------------+ -| RecycleCount | integer | Number of times recycled | -+-----------------+----------+-----------------------------------------+ -| InitialWrite | datetime | When Volume first written | -+-----------------+----------+-----------------------------------------+ -| ScratchPoolId | integer | Id of Scratch Pool | -+-----------------+----------+-----------------------------------------+ -| RecyclePoolId | integer | Pool ID where to recycle Volume | -+-----------------+----------+-----------------------------------------+ -| Comment | blob | User text field | -+-----------------+----------+-----------------------------------------+ - -The **Volume** table (internally referred to as the Media table) -contains one entry for each volume, that is each tape, cassette (8mm, -DLT, DAT, …), or file on which information is or was backed up. There is -one Volume record created for each of the NumVols specified in the Pool -resource record. +The **Media** table contains one entry for each volume, that is each tape +or file on which information is or was backed up. There is one volume record +created for each of the NumVols specified in the Pool resource record. + ++------------------+-----------+-----------------------------------------+ +| Column Name | Data | Remark | +| | Type | | ++==================+===========+=========================================+ +| MediaId | serial | Primary Key | ++------------------+-----------+-----------------------------------------+ +| VolumeName | text | Volume name | ++------------------+-----------+-----------------------------------------+ +| Slot | integer | Autochanger Slot number or zero | ++------------------+-----------+-----------------------------------------+ +| PoolId | integer | Link to Pool Record | ++------------------+-----------+-----------------------------------------+ +| MediaType | text | The MediaType supplied by the user | ++------------------+-----------+-----------------------------------------+ +| MediaTypeId | integer | The MediaTypeId | ++------------------+-----------+-----------------------------------------+ +| LabelType | integer | The type of label on the Volume | ++------------------+-----------+-----------------------------------------+ +| FirstWritten | timestamp | Time/date when first written | ++------------------+-----------+-----------------------------------------+ +| LastWritten | timestamp | Time/date when last written | ++------------------+-----------+-----------------------------------------+ +| LabelDate | timestamp | Time/date when tape labeled | ++------------------+-----------+-----------------------------------------+ +| VolJobs | integer | Number of jobs written to this media | ++------------------+-----------+-----------------------------------------+ +| VolFiles | integer | Number of files written to this media | ++------------------+-----------+-----------------------------------------+ +| VolBlocks | integer | Number of blocks written to this media | ++------------------+-----------+-----------------------------------------+ +| VolMounts | integer | Number of time media mounted | ++------------------+-----------+-----------------------------------------+ +| VolBytes | bigint | Number of bytes saved in Job | ++------------------+-----------+-----------------------------------------+ +| VolErrors | integer | Number of errors during Job | ++------------------+-----------+-----------------------------------------+ +| VolWrites | integer | Number of writes to media | ++------------------+-----------+-----------------------------------------+ +| VolCapacityBytes | bigint | Capacity estimate for this volume | ++------------------+-----------+-----------------------------------------+ +| VolStatus | text | | Status of media: | +| | | | Full, Archive, Append, | +| | | Recycle, Read-Only, Disabled, Error, | +| | | Busy | ++------------------+-----------+-----------------------------------------+ +| Enabled | smallint | Whether or not Volume can be written | ++------------------+-----------+-----------------------------------------+ +| Recycle | smallint | Whether or not Bareos can recycle the | +| | | Volumes: Yes, No | ++------------------+-----------+-----------------------------------------+ +| ActionOnPurge | smallint | What happens to a Volume after purging | ++------------------+-----------+-----------------------------------------+ +| VolRetention | bigint | 64 bit seconds until expiration | ++------------------+-----------+-----------------------------------------+ +| VolUseDureation | bigint | 64 bit seconds volume can be used | ++------------------+-----------+-----------------------------------------+ +| MaxVolJobs | integer | maximum jobs to put on Volume | ++------------------+-----------+-----------------------------------------+ +| MaxVolFiles | integer | maximume EOF marks to put on Volume | ++------------------+-----------+-----------------------------------------+ +| MaxVolBytes | bigint | Maximum bytes to put on this media | ++------------------+-----------+-----------------------------------------+ +| InChanger | smallint | Whether or not Volume in autochanger | ++------------------+-----------+-----------------------------------------+ +| StorageId | integer | Storage record ID | ++------------------+-----------+-----------------------------------------+ +| DeviceId | integer | Device record ID | ++------------------+-----------+-----------------------------------------+ +| MediaAddressing | smallint | Method of addressing media | ++------------------+-----------+-----------------------------------------+ +| VolReadTime | bigint | Time Reading Volume | ++------------------+-----------+-----------------------------------------+ +| VolWriteTime | bigint | Time Writing Volume | ++------------------+-----------+-----------------------------------------+ +| EndFile | integer | End File number of Volume | ++------------------+-----------+-----------------------------------------+ +| EndBlock | bigint | End block number of Volume | ++------------------+-----------+-----------------------------------------+ +| LocationId | integer | Location record ID | ++------------------+-----------+-----------------------------------------+ +| RecycleCount | integer | Number of times recycled | ++------------------+-----------+-----------------------------------------+ +| MinBlockSize | integer | Minimum block size on this media | ++------------------+-----------+-----------------------------------------+ +| MaxBlockSize | integer | Maximum block size on this media | ++------------------+-----------+-----------------------------------------+ +| InitialWrite | timestamp | When Volume first written | ++------------------+-----------+-----------------------------------------+ +| ScratchPoolId | integer | Id of Scratch Pool | ++------------------+-----------+-----------------------------------------+ +| RecyclePoolId | integer | Pool ID where to recycle Volume | ++------------------+-----------+-----------------------------------------+ +| EncryptionKey | text | Key used for encryptoion | ++------------------+-----------+-----------------------------------------+ +| Comment | text | User text field | ++------------------+-----------+-----------------------------------------+ Pool ~~~~ +The **Pool** table contains one entry for each media pool controlled by +Bareos in this database. + +-----------------------+-----------------------+-----------------------+ | Column Name | Data Type | Remark | +=======================+=======================+=======================+ -| PoolId | integer | Primary Key | +| PoolId | serial | Primary Key | +-----------------------+-----------------------+-----------------------+ -| Name | Tinyblob | Pool Name | +| Name | text | Pool Name | +-----------------------+-----------------------+-----------------------+ -| NumVols | Integer | Number of Volumes in | +| NumVols | integer | Number of Volumes in | | | | the Pool | +-----------------------+-----------------------+-----------------------+ -| MaxVols | Integer | Maximum Volumes in | +| MaxVols | integer | Maximum Volumes in | | | | the Pool | +-----------------------+-----------------------+-----------------------+ -| UseOnce | tinyint | Use volume once | +| UseOnce | smallint | Use volume once | +-----------------------+-----------------------+-----------------------+ -| UseCatalog | tinyint | Set to use catalog | +| UseCatalog | smallint | Set to use catalog | +-----------------------+-----------------------+-----------------------+ -| AcceptAnyVolume | tinyint | Accept any volume | +| AcceptAnyVolume | smallint | Accept any volume | | | | from Pool | +-----------------------+-----------------------+-----------------------+ | VolRetention | bigint | 64 bit seconds to | @@ -594,25 +483,25 @@ Pool | MaxVolBytes | bigint | max bytes to write on | | | | Volume | +-----------------------+-----------------------+-----------------------+ -| AutoPrune | tinyint | yes or no for | +| AutoPrune | smallint | yes or no for | | | | autopruning | +-----------------------+-----------------------+-----------------------+ -| Recycle | tinyint | yes or no for | +| Recycle | smallint | yes or no for | | | | allowing auto | | | | recycling of Volume | +-----------------------+-----------------------+-----------------------+ -| ActionOnPurge | tinyint | Default Volume | +| ActionOnPurge | smallint | Default Volume | | | | ActionOnPurge | +-----------------------+-----------------------+-----------------------+ -| PoolType | enum | Backup, Copy, Cloned, | +| PoolType | text | Backup, Copy, Cloned, | | | | Archive, Migration | +-----------------------+-----------------------+-----------------------+ -| LabelType | tinyint | Type of label | +| LabelType | integer | Type of label | | | | ANSI/Bareos | +-----------------------+-----------------------+-----------------------+ -| LabelFormat | Tinyblob | Label format | +| LabelFormat | text | Label format | +-----------------------+-----------------------+-----------------------+ -| Enabled | tinyint | Whether or not Volume | +| Enabled | smallint | Whether or not Volume | | | | can be written | +-----------------------+-----------------------+-----------------------+ | ScratchPoolId | integer | Id of Scratch Pool | @@ -622,6 +511,10 @@ Pool +-----------------------+-----------------------+-----------------------+ | NextPoolId | integer | Pool ID of next Pool | +-----------------------+-----------------------+-----------------------+ +| MinBlockSize | integer | Pool ID of next Pool | ++-----------------------+-----------------------+-----------------------+ +| MaxBlockSize | integer | Pool ID of next Pool | ++-----------------------+-----------------------+-----------------------+ | MigrationHighBytes | bigint | High water mark for | | | | migration | +-----------------------+-----------------------+-----------------------+ @@ -631,59 +524,59 @@ Pool | MigrationTime | bigint | Time before migration | +-----------------------+-----------------------+-----------------------+ -The **Pool** table contains one entry for each media pool controlled by -Bareos in this database. One media record exists for each of the NumVols -contained in the Pool. The PoolType is a Bareos defined keyword. The -MediaType is defined by the administrator, and corresponds to the -MediaType specified in the Director’s Storage definition record. The -CurrentVol is the sequence number of the Media record for the current -volume. +In the **Media** table one or more records exist +for each of the Volumes contained in the Pool. The MediaType is defined +by the administrator, and corresponds to the MediaType specified in the +Director’s Storage definition record. Client ~~~~~~ +The **Client** table contains one entry for each machine backed up by +Bareos in this database. Normally the Name is a fully qualified domain +name. + +---------------+-----------+-------------------------------------+ | Column Name | Data Type | Remark | +===============+===========+=====================================+ -| ClientId | integer | Primary Key | +| ClientId | serial | Primary Key | +---------------+-----------+-------------------------------------+ -| Name | TinyBlob | File Services Name | +| Name | text | File Services Name | +---------------+-----------+-------------------------------------+ -| UName | TinyBlob | uname -a from Client (not yet used) | +| UName | text | uname -a from Client (not yet used) | +---------------+-----------+-------------------------------------+ -| AutoPrune | tinyint | yes or no for autopruning | +| AutoPrune | smallint | yes or no for autopruning | +---------------+-----------+-------------------------------------+ | FileRetention | bigint | 64 bit seconds to retain Files | +---------------+-----------+-------------------------------------+ | JobRentention | bigint | 64 bit seconds to retain Job | +---------------+-----------+-------------------------------------+ -The **Client** table contains one entry for each machine backed up by -Bareos in this database. Normally the Name is a fully qualified domain -name. - Storage ~~~~~~~ +The **Storage** table contains one entry for each Storage used. + +-------------+-----------+---------------------------------+ | Column Name | Data Type | Remark | +=============+===========+=================================+ -| StorageId | integer | Unique Id | +| StorageId | serial | Unique Id | +-------------+-----------+---------------------------------+ -| Name | tinyblob | Resource name of Storage device | +| Name | text | Resource name of Storage device | +-------------+-----------+---------------------------------+ -| AutoChanger | tinyint | Set if it is an autochanger | +| AutoChanger | integer | Set if it is an autochanger | +-------------+-----------+---------------------------------+ -The **Storage** table contains one entry for each Storage used. - Counter ~~~~~~~ +The **Counter** table contains one entry for each permanent counter +defined by the user. + +--------------+-----------+-----------------------------+ | Column Name | Data Type | Remark | +==============+===========+=============================+ -| Counter | tinyblob | Counter name | +| Counter | serial | Counter name | +--------------+-----------+-----------------------------+ | MinValue | integer | Start/Min value for counter | +--------------+-----------+-----------------------------+ @@ -691,108 +584,107 @@ Counter +--------------+-----------+-----------------------------+ | CurrentValue | integer | Current counter value | +--------------+-----------+-----------------------------+ -| WrapCounter | tinyblob | Name of another counter | +| WrapCounter | text | Name of another counter | +--------------+-----------+-----------------------------+ -The **Counter** table contains one entry for each permanent counter -defined by the user. - Log ~~~ +The **Log** table contains a log of all Job output. + +-------------+-----------+------------------------------+ | Column Name | Data Type | Remark | +=============+===========+==============================+ -| LogIdId | integer | Primary Key | +| LogId | serial | Primary Key | +-------------+-----------+------------------------------+ | JobId | integer | Points to Job record | +-------------+-----------+------------------------------+ -| Time | datetime | Time/date log record created | +| Time | timestamp | Time/date log record created | +-------------+-----------+------------------------------+ -| LogText | blob | Log text | +| LogText | text | Log text | +-------------+-----------+------------------------------+ -The **Log** table contains a log of all Job output. - Location ~~~~~~~~ +The **Location** table defines where a Volume is physically. + +-------------+-----------+-----------------------------------+ | Column Name | Data Type | Remark | +=============+===========+===================================+ -| LocationId | integer | Primary Key | +| LocationId | serial | Primary Key | +-------------+-----------+-----------------------------------+ -| Location | tinyblob | Text defining location | +| Location | text | Text defining location | +-------------+-----------+-----------------------------------+ | Cost | integer | Relative cost of obtaining Volume | +-------------+-----------+-----------------------------------+ -| Enabled | tinyint | Whether or not Volume is enabled | +| Enabled | smallint | Whether or not Volume is enabled | +-------------+-----------+-----------------------------------+ -The **Location** table defines where a Volume is physically. - LocationLog ~~~~~~~~~~~ -+-----------+----------+-----------------------------------------------+ -| Column | Data | Remark | -| Name | Type | | -+===========+==========+===============================================+ -| LocLogId | integer | Primary Key | -+-----------+----------+-----------------------------------------------+ -| Date | datetime | Time/date log record created | -+-----------+----------+-----------------------------------------------+ -| MediaId | integer | Points to Media record | -+-----------+----------+-----------------------------------------------+ -| LocationI | integer | Points to Location record | -| d | | | -+-----------+----------+-----------------------------------------------+ -| NewVolSta | integer | enum: Full, Archive, Append, Recycle, Purged | -| tus | | Read-only, Disabled, Error, Busy, Used, | -| | | Cleaning | -+-----------+----------+-----------------------------------------------+ -| Enabled | tinyint | Whether or not Volume is enabled | -+-----------+----------+-----------------------------------------------+ - The **LocationLog** table contains a log of all Job output. ++-------------+-----------+-----------------------------------------------+ +| Column | Data | Remark | +| Name | Type | | ++=============+===========+===============================================+ +| LocLogId | serial | Primary Key | ++-------------+-----------+-----------------------------------------------+ +| Date | timestamp | Time/date log record created | ++-------------+-----------+-----------------------------------------------+ +| Comment | text | Time/date log record created | ++-------------+-----------+-----------------------------------------------+ +| MediaId | integer | Points to Media record | ++-------------+-----------+-----------------------------------------------+ +| LocationId | integer | Points to Location record | ++-------------+-----------+-----------------------------------------------+ +| NewVolStaus | integer | enum: Full, Archive, Append, Recycle, Purged | +| | | Read-only, Disabled, Error, Busy, Used, | +| | | Cleaning | ++-------------+-----------+-----------------------------------------------+ +| Enabled | tinyint | Whether or not Volume is enabled | ++-------------+-----------+-----------------------------------------------+ + Version ~~~~~~~ +The **Version** table defines the Bareos database version number. Bareos +checks this number before reading the database to ensure that it is +compatible with the Bareos binary file. + +-------------+-----------+-------------+ | Column Name | Data Type | Remark | +=============+===========+=============+ | VersionId | integer | Primary Key | +-------------+-----------+-------------+ -The **Version** table defines the Bareos database version number. Bareos -checks this number before reading the database to ensure that it is -compatible with the Bareos binary file. - BaseFiles ~~~~~~~~~ +The **BaseFiles** table contains all the File references for a +particular JobId that point to a Base file. + +-------------+-----------+-------------------+ | Column Name | Data Type | Remark | +=============+===========+===================+ -| BaseId | integer | Primary Key | -+-------------+-----------+-------------------+ -| BaseJobId | integer | JobId of Base Job | +| BaseId | serial | Primary Key | +-------------+-----------+-------------------+ | JobId | integer | Reference to Job | +-------------+-----------+-------------------+ -| FileId | integer | Reference to File | +| FileId | bigint | Reference to File | +-------------+-----------+-------------------+ | FileIndex | integer | File Index number | +-------------+-----------+-------------------+ +| BaseJobId | integer | JobId of Base Job | ++-------------+-----------+-------------------+ -The **BaseFiles** table contains all the File references for a -particular JobId that point to a Base file – i.e. they were previously -saved and hence were not saved in the current JobId but in BaseJobId -under FileId. FileIndex is the index of the file, and is used for -optimization of Restore jobs to prevent the need to read the FileId -record when creating the in memory tree. This record is not yet -implemented. +For example they were previously saved and hence were not saved in +the current JobId but in BaseJobId under FileId. FileIndex is the +index of the file, and is used for optimization of Restore jobs to +prevent the need to read the FileId record when creating the in +memory tree. This record is not yet implemented. UML Diagram of Database Schema ------------------------------ diff --git a/docs/manuals/source/DeveloperGuide/catalog/schema_bareos_postgresql.puml b/docs/manuals/source/DeveloperGuide/catalog/schema_bareos_postgresql.puml index 924bdee497f..75c330e4957 100644 --- a/docs/manuals/source/DeveloperGuide/catalog/schema_bareos_postgresql.puml +++ b/docs/manuals/source/DeveloperGuide/catalog/schema_bareos_postgresql.puml @@ -75,7 +75,6 @@ HasCache : SMALLINT Reviewed : SMALLINT Comment : TEXT - 'PRIMARY KEY (JobId) }