v0.4.2 Partitions handled in static partitioning trigger function is …

…now a configurable option. See CHANGELOG and docs for more info.

CHANGELOG

@@ -1,3 +1,11 @@
+0.4.2
+-- The static partitioning trigger function can now handle partitions based on the configured premake value. For example, the default premake value is 4 so it can now handle data for the current partition, 4 previous partitions and 4 future partitions. Changing the premake value will cause the trigger function to be changed appropriately the next time a partition is automatically created. Except for initial setup, at no time does the automated partitioning system create old partitions (see the create_prev_* functions if you need to do this). If you change the premake value and there is no previous partition for it to put data in, it will go to the parent table.
+-- create_parent() now accounts for the new static partitioning rules. For time-static, it will create the current partition as well as previous and future partitions equal to the configured premake number (default premake being 4, you will end up with 9 partitions). For id-static, it will only create previous partitions if the resulting rules handle id values greater than zero. So if you're starting from zero you will only have future partitions created, and no previous.
+-- Constraint now ensures that premake value is greater than zero.
+-- create_parent() now ensures interval value for serial partitioning is greater than zero.
+-- Much more extensive pgTAP tests.
+
+
 0.4.1
 -- Changed the privilege management system to apply the current parent's privileges only to new child tables at the time they're created. No longer re-applies privileges to existing child tables. When partition sets grew large, this was causing serious performance problems and was too expensive an operation to run every time a child was created.
 -- Dropped apply_grants() function. New child table privileges are now managed by the partition creation functions themselves.

META.json

@@ -1,7 +1,7 @@
 {
     "name": "pg_partman",
     "abstract": "Extension to manage partitioned tables by time or ID",
-    "version": "0.4.1",
+    "version": "0.4.2",
     "maintainer": [ 
         "Keith Fiske <keith@omniti.com>"
     ],
@@ -20,9 +20,9 @@
     },
     "provides": {
         "pg_partmaint": {
-            "file": "sql/pg_partman--0.4.1.sql",
+            "file": "sql/pg_partman--0.4.2.sql",
             "docfile": "doc/pg_partman.md",
-            "version": "0.4.1",
+            "version": "0.4.2",
             "abstract": "Extension to manage partitioned tables by time or ID"
         }
     },

README.md

@@ -5,9 +5,10 @@ pg_partman is an extension to create and manage both time-based and serial-based
 
 INSTALLATION
 ------------
-Recommended: pg_jobmon (>=1.0.0). PG Job Monitor will automatically be used if it is installed.
+Recommended: pg_jobmon (>=1.0.0). PG Job Monitor will automatically be used if it is installed.  
+https://github.com/omniti-labs/pg_jobmon
 
-In directory where you downloaded mimeo to run
+In directory where you downloaded pg_partman to run
 
     make
     make install
@@ -38,7 +39,7 @@ If you're looking to do time-based partitioning, and will only be inserting new
 
     SELECT part.create_parent('test.part_test', 'col3', 'time-static', 'daily');
 
-This will turn your table into a parent table and premake 4 future partitions. To make new partitions for time-based partitioning, use the run_maintenance() function. Ideally, you'd run this as a cronjob to keep new partitions premade in preparation of new data.
+This will turn your table into a parent table and premake 4 future partitions and also make 4 past partitions. To make new partitions for time-based partitioning, use the run_maintenance() function. Ideally, you'd run this as a cronjob to keep new partitions premade in preparation of new data.
 
 This should be enough to get you started. Please see the pg_partman.md file in the doc folder for more information on the types of partitioning supported and what the parameters in the create_parent() function mean. 
 

doc/pg_partman.md

@@ -17,17 +17,17 @@ Functions
 ---------
 A superuser must be used to run these functions in order to set privileges & ownership properly in all cases. All are set with SECURITY DEFINER, so if you cannot have a superuser running them just assign a superuser role as the owner. 
 
-*create_parent(p_parent_table text, p_control text, p_type part.partition_type, p_interval text, p_premake int DEFAULT 4, p_debug boolean DEFAULT false)*
+*create_parent(p_parent_table text, p_control text, p_type text, p_interval text, p_premake int DEFAULT 4, p_debug boolean DEFAULT false)*
  * Main function to create a partition set with one parent table and inherited children. Parent table must already exist. Please apply all defaults, indexes, constraints, privileges & ownership to parent table so they will propagate to children.
  * An ACCESS EXCLUSIVE lock is taken on the parent table during the running of this function. No data is moved when running this function, so lock should be brief.
  * p_parent_table - the existing parent table. MUST be schema qualified, even if in public schema.
  * p_control - the column that the partitioning will be based on. Must be a time or integer based column.
  * p_type - one of 4 values to set the partitioning type that will be used
 
- > **time-static** - Trigger function inserts only into specifically named partitions (handles data for current partition, 2 partitions ahead and 1 behind).  Cannot handle inserts to parent table outside the hard-coded time window. Function is kept up to date by run_maintenance() function. Ideal for high TPS tables that get inserts of new data only.  
+ > **time-static** - Trigger function inserts only into specifically named partitions. The the number of partitions managed behind and ahead of the current one is determined by the **premake** config value (default of 4 means data for 4 previous and 4 future partitions are handled automatically). *Beware setting the premake value too high as that will lessen the efficiency of this partitioning method.* Inserts to parent table outside the hard-coded time window will go to the parent. Trigger function is kept up to date by run_maintenance() function. Ideal for high TPS tables that get inserts of new data only.  
  > **time-dynamic** - Trigger function can insert into any child partition based on the value of the control column. More flexible but not as efficient as time-static. Be aware that if the appropriate partition doesn't yet exist for the data inserted, data gets inserted to the parent.  
- > **id-static** - Same functionality as time-static but for a numeric range instead of time. When the id value has reached 50% of the max value for that partition, it will automatically create the next partition in sequence if it doesn't yet exist. Does NOT require run_maintenance() function to create new partitions.  
- > **id-dynamic** - Same functionality and limitations as time-dynamic but for a numeric range instead of time. Uses same 50% rule as id-static to create future partitions. Does NOT require run_maintenance() function to create new partitions.  
+ > **id-static** - Same functionality and use of the premake value as time-static but for a numeric range instead of time. When the id value has reached 50% of the max value for that partition, it will automatically create the next partition in sequence if it doesn't yet exist. Does NOT require run_maintenance() function to create new partitions. Only supports id values greater than or equal to zero.
+ > **id-dynamic** - Same functionality and limitations as time-dynamic but for a numeric range instead of time. Uses same 50% rule as id-static to create future partitions. Does NOT require run_maintenance() function to create new partitions. Only supports id values greater than or equal to zero.
 
  * p_interval - the time or numeric range interval for each partition. Supported values are:
 
@@ -39,9 +39,9 @@ A superuser must be used to run these functions in order to set privileges & own
  > **hourly** - One partition per hour  
  > **half-hour** - One partition per 30 minute interval on the half-hour (1200, 1230)  
  > **quarter-hour** - One partition per 15 minute interval on the quarter-hour (1200, 1215, 1230, 1245)  
- > **<integer>** - For ID based partitions, the integer value range of the ID that should be set per partition. This is the actual integer value, not text values like time-based partitioning.  
+ > **<integer>** - For ID based partitions, the integer value range of the ID that should be set per partition. This is the actual integer value, not text values like time-based partitioning. Must be greater than zero.
 
- * p_premake - is how many additional partitions to always stay ahead of the current partition. Default value is 4. This will keep at minimum 5 partitions made, including the current one. For example, if today was Sept 6, 2012, and premake was set to 4 for a daily partition, then partitions would be made for the 6th as well as the 7th, 8th, 9th and 10th. 
+ * p_premake - is how many additional partitions to always stay ahead of the current partition. Default value is 4. This will keep at minimum 5 partitions made, including the current one. For example, if today was Sept 6, 2012, and premake was set to 4 for a daily partition, then partitions would be made for the 6th as well as the 7th, 8th, 9th and 10th. As stated above, this value also determines how many partitions outside of the current one the static partitioning trigger function will handle.
  * p_debug - turns on additional debugging information (not yet working).
 
 *run_maintenance()*
@@ -85,17 +85,17 @@ A superuser must be used to run these functions in order to set privileges & own
  * p_keep_index - optional parameter to tell partman whether to keep or drop the indexes of the child table when it is uninherited. TRUE means the indexes will be kept; FALSE means all indexes will be dropped. This function will just use the value configured in **part_config** if not explicitly set. This option is ignored if p_keep_table is set to FALSE.
 
 *reapply_privileges(p_parent_table text)*
- * This function is used to reapply ownership & grants on all child tables based on what the parent table has set.
+ * This function is used to reapply ownership & grants on all child tables based on what the parent table has set. 
  * Privileges that the parent table has will be granted to all child tables and privilges that the parent does not have will be revoked (with CASCADE).
  * Privilges that are checked for are SELECT, INSERT, UPDATE, DELETE, TRUNCATE, REFERENCES, & TRIGGER.
  * Be aware that for large partition sets, this can be a very long running operation and is why it was made into a separate function to run independently. Only privileges that are different between the parent & child are applied, but it still has to do system catalog lookups and comparisons for every single child partition and all individual privileges on each.
- * p_parent_table - parent table of the partition set. Must match a parent table name already configured in pg_partman.
+ * p_parent_table - parent table of the partition set. Must be schema qualified and match a parent table name already configured in pg_partman.
 
 Tables
 ------
 *part_config*  
     Stores all configuration data for partition sets mananged by the extension. The only columns in this table that should ever need to be manually changed are
-    **retention**, **retention_keep_table**, & **retention_keep_index** to set the partition set's retention policy or **premake** to change the default.   
+    **retention**, **retention_keep_table**, & **retention_keep_index** to configure the partition set's retention policy or **premake** to change the default.   
     The rest are managed by the extension itself and should not be changed unless absolutely necessary.
 
     parent_table            - Parent table of the partition set
@@ -104,6 +104,7 @@ Tables
                               Must be a value that can either be cast to the interval or bigint data types.
     control                 - Column used as the control for partition constraints. Must be a time or integer based column.
     premake                 - How many partitions to keep pre-made ahead of the current partition. Default is 4.
+                              Also manages number of partitions handled by static partitioning method. See create_parent() function for more info.
     retention               - Text type value that determines how old the data in a child partition can be before it is dropped. 
                               Must be a value that can either be cast to the interval or bigint data types. 
                               Leave this column NULL (the default) to always keep all child partitions. See **About** section for more info.

pg_partman.control

@@ -1,3 +1,3 @@
-default_version = '0.4.1'
+default_version = '0.4.2'
 comment = 'Extension to manage partitioned tables by time or ID'
 relocatable = false

sql/functions/create_id_function.sql

@@ -6,10 +6,6 @@ CREATE FUNCTION create_id_function(p_parent_table text, p_current_id bigint) RET
     AS $$
 DECLARE
 
-v_1st_partition_name            text;
-v_1st_partition_id              bigint;
-v_2nd_partition_name            text;
-v_2nd_partition_id              bigint;
 v_control                       text;
 v_current_partition_name        text;
 v_current_partition_id          bigint;
@@ -18,16 +14,17 @@ v_final_partition_id            bigint;
 v_job_id                        bigint;
 v_jobmon_schema                 text;
 v_last_partition                text;
+v_next_partition_id             bigint;
+v_next_partition_name           text;
 v_old_search_path               text;
 v_part_interval                 bigint;
 v_premake                       int;
-v_prev_partition_name           text;
 v_prev_partition_id             bigint;
+v_prev_partition_name           text;
 v_step_id                       bigint;
 v_trig_func                     text;
 v_type                          text;
 
-
 BEGIN
 
 SELECT nspname INTO v_jobmon_schema FROM pg_namespace n, pg_extension e WHERE e.extname = 'pg_jobmon' AND e.extnamespace = n.oid;
@@ -46,25 +43,23 @@ SELECT type
     , control
     , premake
     , last_partition
+INTO v_type
+    , v_part_interval
+    , v_control
+    , v_premake
+    , v_last_partition
 FROM @extschema@.part_config 
 WHERE parent_table = p_parent_table
-AND (type = 'id-static' OR type = 'id-dynamic')
-INTO v_type, v_part_interval, v_control, v_premake, v_last_partition;
+AND (type = 'id-static' OR type = 'id-dynamic');
+
 IF NOT FOUND THEN
     RAISE EXCEPTION 'ERROR: no config found for %', p_parent_table;
 END IF;
 
 IF v_type = 'id-static' THEN
     v_current_partition_id := p_current_id - (p_current_id % v_part_interval);
-    v_prev_partition_id := v_current_partition_id - v_part_interval;    
-    v_1st_partition_id := v_current_partition_id + v_part_interval;
-    v_2nd_partition_id := v_1st_partition_id + v_part_interval;
-    v_final_partition_id := v_2nd_partition_id + v_part_interval;
-
-    v_prev_partition_name := p_parent_table || '_p' || v_prev_partition_id::text;
+    v_next_partition_id := v_current_partition_id + v_part_interval;
     v_current_partition_name := p_parent_table || '_p' || v_current_partition_id::text;
-    v_1st_partition_name := p_parent_table || '_p' || v_1st_partition_id::text;
-    v_2nd_partition_name := p_parent_table || '_p' || v_2nd_partition_id::text;
 
     v_trig_func := 'CREATE OR REPLACE FUNCTION '||p_parent_table||'_part_trig_func() RETURNS trigger LANGUAGE plpgsql AS $t$ 
         DECLARE
@@ -74,21 +69,28 @@ IF v_type = 'id-static' THEN
             v_next_partition_name   text;         
         BEGIN
         IF TG_OP = ''INSERT'' THEN 
-            IF NEW.'||v_control||' >= '||v_current_partition_id||' AND NEW.'||v_control||' < '||v_1st_partition_id|| ' THEN 
-                INSERT INTO '||v_current_partition_name||' VALUES (NEW.*); 
-            ELSIF NEW.'||v_control||' >= '||v_1st_partition_id||' AND NEW.'||v_control||' < '||v_2nd_partition_id|| ' THEN 
-                INSERT INTO '||v_1st_partition_name||' VALUES (NEW.*); 
-            ELSIF NEW.'||v_control||' >= '||v_2nd_partition_id||' AND NEW.'||v_control||' < '||quote_literal(v_final_partition_id)|| ' THEN 
-                INSERT INTO '||v_2nd_partition_name||' VALUES (NEW.*);
-            ';
-            -- If the first partition's function, don't have rule for previous partition
+            IF NEW.'||v_control||' >= '||v_current_partition_id||' AND NEW.'||v_control||' < '||v_next_partition_id|| ' THEN 
+                INSERT INTO '||v_current_partition_name||' VALUES (NEW.*); ';
+
+        FOR i IN 1..v_premake LOOP
+            v_prev_partition_id := v_current_partition_id - (v_part_interval * i);
+            v_next_partition_id := v_current_partition_id + (v_part_interval * i);
+            v_final_partition_id := v_next_partition_id + v_part_interval;
+            v_prev_partition_name := p_parent_table || '_p' || v_prev_partition_id::text;
+            v_next_partition_name := p_parent_table || '_p' || v_next_partition_id::text;
+            -- Only make previous partitions if they're starting above zero
             IF v_prev_partition_id >= 0 THEN
-            v_trig_func := v_trig_func ||'ELSIF NEW.'||v_control||' >= '||v_prev_partition_id||' AND NEW.'||v_control||' < '||v_current_partition_id|| ' THEN 
-                INSERT INTO '||v_prev_partition_name||' VALUES (NEW.*);
-            ';
+                v_trig_func := v_trig_func ||'
+            ELSIF NEW.'||v_control||' >= '||v_prev_partition_id||' AND NEW.'||v_control||' < '||v_prev_partition_id + v_part_interval|| ' THEN 
+                INSERT INTO '||v_prev_partition_name||' VALUES (NEW.*); ';
             END IF;
+            v_trig_func := v_trig_func ||'
+            ELSIF NEW.'||v_control||' >= '||v_next_partition_id||' AND NEW.'||v_control||' < '||v_final_partition_id|| ' THEN 
+                INSERT INTO '||v_next_partition_name||' VALUES (NEW.*); ';
+        END LOOP;
 
-            v_trig_func := v_trig_func ||'ELSE
+        v_trig_func := v_trig_func ||'
+            ELSE
                 RETURN NEW;
             END IF;
             v_current_partition_id := NEW.'||v_control||' - (NEW.'||v_control||' % '||v_part_interval||');
@@ -105,11 +107,10 @@ IF v_type = 'id-static' THEN
         RETURN NULL; 
         END $t$;';
 
-    --RAISE NOTICE 'v_trig_func: %',v_trig_func;
     EXECUTE v_trig_func;
 
     IF v_jobmon_schema IS NOT NULL THEN
-        PERFORM update_step(v_step_id, 'OK', 'Added function for current id interval: '||v_current_partition_id||' to '||v_1st_partition_id-1);
+        PERFORM update_step(v_step_id, 'OK', 'Added function for current id interval: '||v_current_partition_id||' to '||v_final_partition_id-1);
     END IF;
 
 ELSIF v_type = 'id-dynamic' THEN
@@ -157,7 +158,6 @@ ELSIF v_type = 'id-dynamic' THEN
         RETURN NULL; 
         END $t$;';
 
---    RAISE NOTICE 'v_trig_func: %',v_trig_func;
     EXECUTE v_trig_func;
 
     IF v_jobmon_schema IS NOT NULL THEN