Merge 72b0db9 into a001ea7

.editorconfig

@@ -0,0 +1,9 @@
+root = true
+
+[*]
+indent_style = space
+indent_size = 2
+end_of_line = lf
+charset = utf-8
+trim_trailing_whitespace = true
+insert_final_newline = true

.npmignore

@@ -4,5 +4,4 @@ coverage/
 *.tgz
 .nyc_output/
 docs/
-CHANGELOG.md
-README.md
+.editorconfig

.travis.yml

@@ -8,8 +8,9 @@ language: node_js
 node_js:
   - "12"
   - "10"
-  - "8"  
+  - "8"
 before_script:
   - psql -c 'create database pgboss' -U postgres
+  - psql -c 'create extension pgcrypto' -d pgboss -U postgres
 script:
   - npm run forcover

README.md

@@ -14,11 +14,11 @@ async function readme() {
   boss.on('error', error => console.error(error));
 
   await boss.start();
-  
+
   const queue = 'some-queue';
 
   let jobId = await boss.publish(queue, { param1: 'foo' })
-  
+
   console.log(`created job in queue ${queue}: ${jobId}`);
 
   await boss.subscribe(queue, someAsyncJobHandler);
@@ -27,16 +27,16 @@ async function readme() {
 async function someAsyncJobHandler(job) {
   console.log(`job ${job.id} received with data:`);
   console.log(JSON.stringify(job.data));
-    
+
   await doSomethingAsyncWithThis(job.data);
 }
 ```
 
-pg-boss is a job queue built in Node.js on top of PostgreSQL in order to provide background job processing and reliable asynchronous execution to Node.js applications.
+pg-boss is a job queue built in Node.js on top of PostgreSQL in order to provide background processing and reliable asynchronous execution to Node.js applications.
 
 Why would you consider using this queue over others? pg-boss is actually a light abstraction over features added in PostgreSQL 9.5
 (specifically [SKIP LOCKED](http://blog.2ndquadrant.com/what-is-select-skip-locked-for-in-postgresql-9-5) and upserts)
-which significantly enhanced its ability to act as a reliable, distributed message queue. I wrote this to remove a dependency on Redis (via the kue package), consolidating systems I have to support in production as well as upgrading to guaranteed message processing (hint: [Redis persistence docs](https://redis.io/topics/persistence#ok-so-what-should-i-use)). 
+which significantly enhanced its ability to act as a reliable, distributed message queue. I wrote this to remove a dependency on Redis (via the kue package), consolidating systems I have to support in production as well as upgrading to guaranteed message processing (hint: [Redis persistence docs](https://redis.io/topics/persistence#ok-so-what-should-i-use)).
 
 This will likely cater the most to teams already familiar with the simplicity of relational database semantics and operations (querying and backups, for example).
 
@@ -45,12 +45,14 @@ This will likely cater the most to teams already familiar with the simplicity of
 * Delayed jobs
 * Job retries (opt-in exponential backoff)
 * Job throttling (unique jobs, rate limiting and/or debouncing)
-* Job batching for high volume use cases 
+* Job batching for high volume use cases
 * Backpressure-compatible subscriptions
 * Configurable job concurrency
 * Distributed and/or clustered workers
 * Completion subscriptions to support orchestrations/sagas
 * On-demand job fetching and completion for external integrations (such as web APIs)
+* Multi-master capable using tools such as Kubernetes ReplicaSets
+* Direct table access for bulk loading via COPY or other advanced usage
 * Automatic provisioning of required storage into a dedicated schema
 * Automatic monitoring for expired jobs
 * Automatic archiving for completed jobs

docs/configuration.md

@@ -1,31 +1,31 @@
-Configuration
-=============
+# Configuration <!-- omit in toc -->
 
 pg-boss can be customized using configuration options when an instance is created (the constructor argument), during publishing as well as subscribing.
 
 <!-- TOC -->
 
 - [Constructor Options](#constructor-options)
-    - [Database options](#database-options)
-    - [Job monitor options](#job-monitor-options)
-    - [Job creation options](#job-creation-options)
-    - [Job fetch options](#job-fetch-options)
-    - [Job expiration options](#job-expiration-options)
-    - [Job archive options](#job-archive-options)
+  - [Database options](#database-options)
+  - [Queue options](#queue-options)
+  - [Maintenance options](#maintenance-options)
+    - [Archive completed jobs](#archive-completed-jobs)
+    - [Delete archived jobs](#delete-archived-jobs)
+    - [Maintenance interval](#maintenance-interval)
 - [Publish Options](#publish-options)
-    - [Delayed jobs](#delayed-jobs)
-    - [Unique jobs](#unique-jobs)
-    - [Throttled jobs](#throttled-jobs)
-    - [Job retries](#job-retries)
-    - [Job expiration](#job-expiration)
+  - [Delayed jobs](#delayed-jobs)
+  - [Unique jobs](#unique-jobs)
+  - [Throttled jobs](#throttled-jobs)
+  - [Job retries](#job-retries)
+  - [Job expiration](#job-expiration)
+  - [Job retention](#job-retention)
 - [Subscribe Options](#subscribe-options)
 
 <!-- /TOC -->
 
 ## Constructor Options
 
 The constructor accepts either a string or an object.  If a string is used, it's interpreted as a Postgres connection string.
-Since passing only a connection string is intended to be for convenience, you can't set any other options.   
+Since passing only a connection string is intended to be for convenience, you can't set any other options.
 
 ### Database options
 
@@ -43,58 +43,55 @@ Since passing only a connection string is intended to be for convenience, you ca
 
 * **connectionString** - string
 
-    PostgresSQL connection string will be parsed and used instead of `host`, `port`, `ssl`, `database`, `user`, `password`.
-    Based on the [pg](https://github.com/brianc/node-postgres) package. For example: 
-    
+    PostgreSQL connection string will be parsed and used instead of `host`, `port`, `ssl`, `database`, `user`, `password`.
+    Based on the [pg](https://github.com/brianc/node-postgres) package. For example:
+
     ```js
     const boss = new PgBoss('postgres://user:pass@host:port/database?ssl=require');
     ```
 
 * **poolSize** or **max** - int, defaults to 10
 
     Maximum number of connections that will be shared by all subscriptions in this instance
-    
+
 * **application_name** - string, defaults to "pgboss"
-    
+
 * **db** - object
 
-    Passing an object named db allows you "bring your own database connection".  
+    Passing an object named db allows you "bring your own database connection".
     Setting this option ignores all of the above settings. The interface required for db is a single function called `executeSql` that accepts a SQL string and an optional array of parameters. This should return a promise that resolves an object just like the pg module: a `rows` array with results and `rowCount` property that contains affected records after an update operation.
-    
+
     ```js
     {
       // resolves Promise
-      executeSql(text, [values])    
+      executeSql(text, [values])
     }
     ```
-    
-    This option may be beneficial if you'd like to use an existing database service 
+
+    This option may be beneficial if you'd like to use an existing database service
     with its own connection pool.
-    
-    For example, you may be relying on the cluster module on 
-    a web server, and you'd like to limit the growth of total connections as much as possible. 
+
+    For example, you may be relying on the cluster module on
+    a web server, and you'd like to limit the growth of total connections as much as possible.
 
 * **schema** - string, defaults to "pgboss"
 
     Only alphanumeric and underscore allowed, length: <= 50 characters
 
-### Job monitor options
+### Queue options
+
+* **uuid** - string, defaults to "v1"
+
+    job uuid format used, "v1" or "v4"
 
 * **monitorStateIntervalSeconds** - int, default undefined
 
-    Specifies how often in seconds an instance will fire the `monitor-states` event. Cannot be less than 1.
+    Specifies how often in seconds an instance will fire the `monitor-states` event. Cannot be less than 1.  This is only available
 
 * **monitorStateIntervalMinutes** - int, default undefined
 
     Specifies how often in minutes an instance will fire the `monitor-states` event. Cannot be less than 1. Do not use if using `monitorStateIntervalSeconds`.
 
-### Job creation options
-
-* **uuid** - string, defaults to "v1"
-
-    uuid format used, "v1" or "v4"
-
-### Job fetch options
 * **newJobCheckInterval**, int
 
     interval to check for new jobs in milliseconds, must be >=100
@@ -103,54 +100,74 @@ Since passing only a connection string is intended to be for convenience, you ca
 
     Default: 1. interval to check for new jobs in seconds, must be >=1
 
-When `newJobCheckIntervalSeconds` is specified, `newJobCheckInterval` is ignored.
+> When `newJobCheckIntervalSeconds` is specified, `newJobCheckInterval` is ignored.
+
+### Maintenance options
+
+Maintenance operations include checking active jobs for expiration, archiving completed jobs from the primary job table, and deleting archived jobs from the archive table.
+
+* **noSupervisor**, bool, default undefined
+
+  If this is set to true, maintenance and monitoring operations will not be started during a `start()` after the schema is created.  This is an advanced use case, as bypassing maintenance operations is not something you would want to do under normal circumstances.
+
+#### Archive completed jobs
+
+* **archiveIntervalSeconds**, int
 
-### Job expiration options
-* **expireCheckInterval**, int
+    archive interval in seconds, must be >=1
 
-    interval to expire jobs in milliseconds, must be >=100
+* **archiveIntervalMinutes**, int
 
-* **expireCheckIntervalSeconds**, int
+    archive interval in minutes, must be >=1
 
-    interval to expire jobs in seconds, must be >=1
+* **archiveIntervalHours**, int
 
-* **expireCheckIntervalMinutes**, int
+    archive interval in hours, must be >=1
 
-    Default: 1. interval to expire jobs in minutes, must be >=1
+* **archiveIntervalDays**, int
 
-When `expireCheckIntervalMinutes` is specified, `expireCheckIntervalSeconds` and `expireCheckInterval` are ignored.
+    archive interval in days, must be >=1
 
-When `expireCheckIntervalSeconds` is specified, `expireCheckInterval` is ignored.
+Default: 1 hour.  When jobs become eligible for archive after completion.
 
-### Job archive options
+> When a higher unit is is specified, such as hours, lower unit configuration settings are ignored.
 
-* **archiveCompletedJobsEvery**, string, [PostgreSQL interval](https://www.postgresql.org/docs/9.5/static/datatype-datetime.html#DATATYPE-INTERVAL-INPUT)
+#### Delete archived jobs
 
-    Default: "1 hour".  When jobs become eligible for archive after completion.
+* **deleteIntervalSeconds**, int
 
-* **archiveCheckInterval**, int
+    delete interval in seconds, must be >=1
 
-    interval to archive jobs in milliseconds, must be >=100
+* **deleteIntervalMinutes**, int
 
-* **archiveCheckIntervalSeconds**, int
+    delete interval in minutes, must be >=1
 
-    interval to archive jobs in seconds, must be >=1
+* **deleteIntervalHours**, int
 
-* **archiveCheckIntervalMinutes**, int
+    delete interval in hours, must be >=1
 
-    Default: 60. interval to archive jobs in minutes, must be >=1
+* **deleteIntervalDays**, int
 
-When `archiveCheckIntervalMinutes` is specified, `archiveCheckIntervalSeconds` and `archiveCheckInterval` are ignored.
+    delete interval in days, must be >=1
 
-When `archiveCheckIntervalSeconds` is specified, `archiveCheckInterval` is ignored.
+Default: 7 days  When jobs in the archive table become eligible for deletion.
 
-* **deleteArchivedJobsEvery**, string, [PostgreSQL interval](https://www.postgresql.org/docs/9.5/static/datatype-datetime.html#DATATYPE-INTERVAL-INPUT)
+> When a higher unit is is specified, such as hours, lower unit configuration settings are ignored.
 
-    Default: "7 days".  When jobs in the archive table become eligible for deletion.
+#### Maintenance interval
 
-* **deleteCheckInterval**, int
+* **maintenanceIntervalSeconds**, int
+
+    maintenance interval in seconds, must be >=1
+
+* **maintenanceIntervalMinutes**, int
+
+    Default: 1. interval in minutes, must be >=1
+
+> When `maintenanceIntervalMinutes` is specified, `maintenanceIntervalSeconds` and `maintenanceInterval` are ignored.
+>
+> When `maintenanceIntervalSeconds` is specified, `maintenanceInterval` is ignored.
 
-    interval to delete jobs in milliseconds, must be >=100.   Default: 1 hour
 
 ## Publish Options
 
@@ -172,7 +189,7 @@ When `archiveCheckIntervalSeconds` is specified, `archiveCheckInterval` is ignor
 Only allows 1 job (within the same name) to be queued or active with the same singletonKey.
 
 ```js
-boss.publish('my-job', {}, {singletonKey: '123'}) // resolves a jobId 
+boss.publish('my-job', {}, {singletonKey: '123'}) // resolves a jobId
 boss.publish('my-job', {}, {singletonKey: '123'}) // resolves a null jobId until first job completed
 ```
 
@@ -190,7 +207,7 @@ For example, if you set the `singletonMinutes` to 1, then submit 2 jobs within a
 
 Order of precedence for throttling is least to greatest. For example, if `singletonSeconds` is set, `singletonMinutes` is ignored.
 
-Setting `singletonNextSlot` to true will cause the job to be scheduled to run after the current time slot if and when a job is throttled.  Basically it's debounce with a lousy name atm.  Expect this api to be improved in the future.  
+Setting `singletonNextSlot` to true will cause the job to be scheduled to run after the current time slot if and when a job is throttled.  Basically it's debounce with a lousy name atm.  Expect this api to be improved in the future.
 
 ### Job retries
 
@@ -208,9 +225,39 @@ Setting `singletonNextSlot` to true will cause the job to be scheduled to run af
 
 ### Job expiration
 
-* **expireIn**, string, PostgreSQL interval
+* **expireInSeconds**, number
+
+    How many seconds a job may be in active state before it is failed because of expiration. Must be >=1
+
+* **expireInMinutes**, number
+
+    How many minutes a job may be in active state before it is failed because of expiration. Must be >=1
+
+* **expireInHours**, number
+
+    How many hours a job may be in active state before it is failed because of expiration. Must be >=1
+
+Default: 15 minutes
+
+### Job retention
+
+* **retentionSeconds**, number
+
+    How many seconds a job may be in created state before it becomes eligible to be archived. Must be >=1
+
+* **retentionMinutes**, number
+
+    How many minutes a job may be in created state before it becomes eligible to be archived. Must be >=1
+
+* **retentionHours**, number
+
+    How many hours a job may be in created state before it becomes eligible to be archived. Must be >=1
+
+* **retentionDays**, number
+
+    How many days a job may be in created state before it becomes eligible to be archived. Must be >=1
 
-    Default: 15 minutes. Amount of time a job can be actively executing before it is marked as expired.
+Default: 30 days
 
 ## Subscribe Options