Optimize planning times when hypertables have many chunks #502

cevian · 2018-04-17T18:49:52Z

This planner optimization reduces planning times when a hypertable has many chunks.
It does this by expanding hypertable chunks manually, eliding the expand_inherited_tables
logic used by PG.

Slow planning time were previously seen because expand_inherited_tables expands all chunks of
a hypertable, without regard to constraints present in the query. Then, get_relation_info is
the called on all chunks before constraint exclusion. Getting the statistics an many chunks ends
up being expensive because RelationGetNumberOfBlocks has to open the file for each relation.
This gets even worse under high concurrency.

This logic solves this by expanding only the chunks needed to fulfil the query instead of all chunks.
In effect, it moves chunk exclusion up in the planning process. But, we actually don't use constraint
exclusion here, but rather a variant of range exclusion implemented
by HypertableRestrictInfo.

cevian · 2018-04-17T19:24:15Z

This replaces #471.

RobAtticus

Quick review, will let other take deep dive

RobAtticus · 2018-04-20T23:51:34Z

src/chunk.c

+	chunk_scan_ctx_init(&ctx, hs, NULL);
+
+	/* Abort the scan when the chunk is found */
+	ctx.early_abort = false;


Comment & code don't match, at least I don't think. It appears that you are not ending the scan when the chunk is found.

RobAtticus · 2018-04-20T23:53:33Z

src/plan_expand_hypertable.c

+find_children_oids(HypertableRestrictInfo *hri, Hypertable *ht, LOCKMODE lockmode)
+{
+	/*
+	 * optimization: using the HRI only makes sense if we ar not using all the


nit: ar -> are

RobAtticus · 2018-04-20T23:54:29Z

src/plan_expand_hypertable.c

+							  bool inhparent,
+							  RelOptInfo *rel)
+{
+	RangeTblEntry *rte = rt_fetch(rel->relid, root->parse->rtable);


this indentation seems wrong

RobAtticus · 2018-04-24T15:26:31Z

src/plan_expand_hypertable.c

+	Index		rti = rel->relid;
+	List	   *appinfos = NIL;
+	HypertableRestrictInfo *hri;
+	PlanRowMark *oldrc;


Unused variable (according to GCC on Ubuntu)

actually nevermind, i see it getting used so I'm not sure why it complains.

I guess because it's getting used in Assert and nowhere else it gets stripped out in Release builds. May need to use it in a no-op to get rid of the warning.

goodkiller · 2018-04-25T06:30:34Z

Please fix this ASAP, because I have approx 50 chunks, and queries are very slow. What could be estimated fix date?

erimatnor · 2018-04-25T13:47:05Z

Hi @goodkiller thanks for your interest in this PR. We're in the process of reviewing this new functionality and we aim to get it in for the next release.

You mentioned ~50 chunks for your setup, which doesn't seem like a lot, actually. Wondering if you are experiencing some other issue?

goodkiller · 2018-04-25T14:10:00Z

Hi @erimatnor
I was partitioned data by every day and got data around 3 weeks period of 400 GB sensor readings. pgbench result was around 20TPS/s. After i truncate all data and created partitions week precision and same amount of data then it performs well so far, TPS is around 65. BUT, I have to migrate data since from 2015... and I afraid that this kind bug should be fixed to perform in future as well.

RobAtticus · 2018-04-25T21:53:40Z

@erimatnor @cevian Benchmark numbers look good to me. Big improvement for a dataset with 4000+ (600ms -> 36ms) chunks and a more modest improvement for one with only about 6 chunks (6.6ms -> 5.9-6ms).

So even if it doesn't do a ton for the low end, it's not hurting performance and is a big boon for the many chunks case. Pending the fixes I suggested, it has my approval.

erimatnor

Overall, I think the optimization is good. A bunch of nits and suggestions though.

erimatnor · 2018-04-20T12:07:19Z

src/plan_expand_hypertable.c

+	 */
+	hri = hypertable_restrict_info_create(rel, ht);
+	hypertable_restrict_info_add(hri, root, restrictinfo);
+	inhOIDs = find_children_oids(hri, ht, lockmode);


inhOIDs -> inh_oids

erimatnor · 2018-04-20T12:07:41Z

src/plan_expand_hypertable.c

+
+	foreach(l, inhOIDs)
+	{
+		Oid			childOID = lfirst_oid(l);


childOID -> child_oid

erimatnor · 2018-04-20T12:08:02Z

src/plan_expand_hypertable.c

+		Oid			childOID = lfirst_oid(l);
+		Relation	newrelation;
+		RangeTblEntry *childrte;
+		Index		childRTindex;


childRTIndex -> child_rtindex

erimatnor · 2018-04-20T12:08:28Z

src/plan_expand_hypertable.c

+{
+	RangeTblEntry *rte = rt_fetch(rel->relid, root->parse->rtable);
+	List	   *inhOIDs;
+	Oid			parentOID = relationObjectId;


parentOID -> parent_oid

erimatnor · 2018-04-20T12:08:42Z

src/plan_expand_hypertable.c

+							  RelOptInfo *rel)
+{
+	RangeTblEntry *rte = rt_fetch(rel->relid, root->parse->rtable);
+	List	   *inhOIDs;


inhOIDs -> inh_oids

erimatnor · 2018-04-26T12:24:52Z

src/plan_expand_hypertable.c

+																	 lockmode));
+		return result;
+	}
+	else


unnecessary else clause.

I'd do like suggested above.

erimatnor · 2018-04-26T12:29:41Z

src/hypertable_restrict_info.c

+	{
+		DimensionRestrictInfo *dri = dimension_restrict_info_create(&ht->space->dimensions[i]);
+
+		res->diminson_restriction[AttrNumberGetAttrOffset(ht->space->dimensions[i].column_attno)] = dri;


Ohh, I see you are indexing by column_attno instead of dimension ID, so the array can be sparse, hence pointer array. Is this ideal/necessary? Imagine a table with 100+ columns (which we've seen) where time is last. That would create a really sparse array.

Is it necessary to optimize getting the restriction from the array by attno? Without this, fetching would only be O(n) with the number of dimensions, but could be optimized with hash table or tree if an issue (which it really wouldn't be unless really large number of dimensions)

I believe this is the most efficient representation of this structure because it is most often accessed by attribute number. There is a max number of attributes in PostgreSQL (1500 or something) and each column only takes the size of a pointer so I don't believe the size here is really an issue. I'd rather make the access as efficient as possible.

I changed the names to be more clear.

erimatnor · 2018-04-26T12:36:10Z

src/planner.c

+		foreach(lc, query->rtable)
+		{
+			RangeTblEntry *rte = lfirst(lc);
+			Hypertable *ht = hypertable_cache_get_entry(hc, rte->relid);


Is this guaranteed to be non-NULL? Maybe add an Assert() to make this clear.

No it can be NULL. plan_expand_hypertable_valid_hypertable handles the NULL case.

erimatnor · 2018-04-26T12:42:27Z

src/planner.c

@@ -323,18 +404,53 @@ timescaledb_set_rel_pathlist(PlannerInfo *root,
 	cache_release(hcache);
 }

+static void
+timescaledb_get_relation_info_hook(PlannerInfo *root,


What is the reasoning between expanding the append relation in this hook? Not saying it is wrong, but it seems non-obvious. At least there should be a comment explaining this, and what this hook function does in general (i.e., it expands the hypertable).

erimatnor · 2018-04-26T12:47:59Z

src/plan_expand_hypertable.h

+ *
+ * Slow planning time were previously seen because `expand_inherited_tables` expands all chunks of
+ * a hypertable, without regard to constraints present in the query. Then, `get_relation_info` is
+ * the called on all chunks before constraint exclusion. Getting the statistics an many chunks ends


then called...

cevian · 2018-04-29T21:19:53Z

@RobAtticus @erimatnor Fixed all your comments (unless I replied directly to the msg)

erimatnor · 2018-05-08T12:28:26Z

src/chunk.c


 	chunk_scan_ctx_foreach_chunk(ctx, chunk_is_complete, 1);

+	return (ctx->data == NIL ? NULL : linitial(ctx->data));


Can we make this function simply a wrapper around ...get_chunk_list?

erimatnor · 2018-05-08T12:31:19Z

src/chunk.c

+	}
+
+	/* Get a list of chunks that each have N matching dimension constraints */
+	chunk_list = chunk_scan_ctx_get_chunk_list(&ctx);


Can't you just iterate the chunk scan context here with your own per-chunk handler instead of first creating a list? Seems you are adding new functionality when the equivalent functionality already exists, iterating information twice and doing unnecessary allocations.

erimatnor · 2018-05-08T12:36:14Z

src/dimension_slice.c

 		return true;
 	}
 	else if (other->fd.range_start > coord &&
 			 other->fd.range_start < to_cut->fd.range_end)
 	{
 		/* Cut "after" the coordinate */
 		to_cut->fd.range_end = other->fd.range_start;
+


Is this a new pgindent thing or why this change?

Yes this seems a pgindent thing

erimatnor · 2018-05-08T12:37:48Z

src/hypertable_restrict_info.c

+	}
+}
+
+bool


erimatnor · 2018-05-08T12:39:19Z

src/hypertable_restrict_info.c

+	{
+		DimensionRestrictInfo *dri = dimension_restrict_info_create(&ht->space->dimensions[i]);
+
+		res->dimension_restriction[AttrNumberGetAttrOffset(ht->space->dimensions[i].column_attno)] = dri;


Still not sure about this sparse array. I think the most common case by far is 1 or 2 dimensions, so lookup by iterating the dimensions shouldn't be much worse than array indexing, at least not in any way that matters. I think it is a lot more common to have many columns, potentially partitioning on a high attribute number, than having lots of dimensions. If this proves a problem in the future, we can optimize with a hashtable or similar.

While I agree a list would probably not be /bad/ I think the sparse array is more efficient because of O(1). Since we may have many clauses, I'm not sure why we wouldn't use this. The memory usage is limited as I mentioned before.

There's really only a benefit of O(1) lookups when you have big data sets and not with one or two elements, which is the common case here. I mean, honestly, most of the time your are creating a sparse array with one single element! (Or, am I missing something?). This seems like over-engineering of an otherwise very simple thing. I wouldn't push back if you had a strong argument here, like showing an important efficiency improvement (e.g., significantly faster planning times). But I think, when in doubt, we should go for simplicity and maintainability of the code with the option of optimizing in the future.

Since this seems like a "won't fix", I guess you strongly believe this is an important efficiency/speed optimization, to the extent that it is worth pushing it through. Thus I won't block the PR on this.

Fixed - made it into a non-sparse array

erimatnor · 2018-05-14T08:25:21Z

src/hypertable_restrict_info.c

+dimension_restrict_info_closed_slices(DimensionRestrictInfoClosed *dri)
+{
+	if (dri->strategy == BTEqualStrategyNumber)
+	{


unnecessary braces

erimatnor · 2018-05-14T08:29:35Z

src/plan_expand_hypertable.c

+
+/* Since baserestrictinfo is not yet set by the planner, we have to derive
+ * it ourselves. It's safe for us to miss some restrict info clauses (this
+ * will just results in more chunks being included) so this does not need


results -> result

erimatnor · 2018-05-14T08:30:59Z

src/plan_expand_hypertable.c

+	List	   *result;
+
+	/*
+	 * optimization: using the HRI only makes sense if we are not using all


Ambiguous comment: Is this optimization done now (doesn't look like it), or is it suggested?

erimatnor · 2018-05-14T08:33:19Z

src/plan_expand_hypertable.c

+	Oid			parent_oid = relation_objectid;
+	ListCell   *l;
+	Relation	oldrelation = heap_open(parent_oid, NoLock);
+	LOCKMODE	lockmode = AccessShareLock;


Why does this need to be a variable? I don't see it set anywhere else.

erimatnor · 2018-05-14T08:34:29Z

src/plan_expand_hypertable.c

+{
+	RangeTblEntry *rte = rt_fetch(rel->relid, root->parse->rtable);
+	List	   *inh_oids;
+	Oid			parent_oid = relation_objectid;


Why this extra variable? Don't see it set anywhere. Is it a name clarity issue? Then why not just use the name for the function parameter?

cevian · 2018-05-16T20:57:00Z

@erimatnor ready for another review

RobAtticus · 2018-05-16T21:08:37Z

Build is broken @cevian

erimatnor

Only a few remaining things.

erimatnor · 2018-05-18T09:02:42Z

src/hypertable_restrict_info.c

+	{
+		DimensionRestrictInfo *dri = dimension_restrict_info_create(&ht->space->dimensions[i]);
+
+		res->dimension_restriction[AttrNumberGetAttrOffset(ht->space->dimensions[i].column_attno)] = dri;


There's really only a benefit of O(1) lookups when you have big data sets and not with one or two elements, which is the common case here. I mean, honestly, most of the time your are creating a sparse array with one single element! (Or, am I missing something?). This seems like over-engineering of an otherwise very simple thing. I wouldn't push back if you had a strong argument here, like showing an important efficiency improvement (e.g., significantly faster planning times). But I think, when in doubt, we should go for simplicity and maintainability of the code with the option of optimizing in the future.

Since this seems like a "won't fix", I guess you strongly believe this is an important efficiency/speed optimization, to the extent that it is worth pushing it through. Thus I won't block the PR on this.

erimatnor · 2018-05-18T09:05:32Z

src/plan_expand_hypertable.c

+	Assert(rti != parse->resultRelation);
+	oldrc = get_plan_rowmark(root->rowMarks, rti);
+	if (oldrc && RowMarkRequiresRowShareLock(oldrc->markType))
+	{


I would skip braces here. Also, non-conforming error message

erimatnor · 2018-05-18T09:16:02Z

src/chunk.c

+
+	chunk_scan_ctx_destroy(&ctx);
+
+	foreach(lc, oid_list)


Why not also do this work (locking) in append_chunk_oid (which is what I meant in previous comment)? You are still iterating twice here and then I presume once more when creating the appendInfos. That's at least three iterations of the same data. Ideally, you'd do all work in one iteration. Any reason not to?

erimatnor

Some nits.

erimatnor · 2018-05-25T10:39:43Z

src/chunk.c

+append_chunk_oid(ChunkScanCtx *scanctx, Chunk *chunk)
+{
+	if (chunk_is_complete(scanctx, chunk))
+	{


This is a bit of a style choice, and not a big issue for a small function, but I tend to favor early exits, in this case:

if (!chunk_is_complete(scanctx, chunk)) return false;

This makes code easier to read because you have less indentation and nesting and do not need go to the end of the function to know if the "negative" case means exit or executing some other code.

erimatnor · 2018-05-25T10:55:51Z

src/hypertable_restrict_info.c

+}
+
+static DimensionRestrictInfo *
+hypertable_restrict_info_get(HypertableRestrictInfo *hri, int attno)


Should the attno parameter be of type AttrNumber?

This planner optimization reduces planning times when a hypertable has many chunks. It does this by expanding hypertable chunks manually, eliding the `expand_inherited_tables` logic used by PG. Slow planning time were previously seen because `expand_inherited_tables` expands all chunks of a hypertable, without regard to constraints present in the query. Then, `get_relation_info` is the called on all chunks before constraint exclusion. Getting the statistics an many chunks ends up being expensive because RelationGetNumberOfBlocks has to open the file for each relation. This gets even worse under high concurrency. This logic solves this by expanding only the chunks needed to fulfil the query instead of all chunks. In effect, it moves chunk exclusion up in the planning process. But, we actually don't use constraint exclusion here, but rather a variant of range exclusion implemented by HypertableRestrictInfo.

We hit a bug in 9.6.5 fixed in 9.6.6 by commit 77cd0dc. Also changed extension is transitioning check to not palloc anything. This is more efficient and probably has slightly less side-effects on bugs like this.

cevian requested review from erimatnor, RobAtticus and davidkohn88 April 17, 2018 18:49

cevian force-pushed the plan_expand_hypertables branch 5 times, most recently from 4c12952 to 931d7ca Compare April 17, 2018 19:20

RobAtticus reviewed Apr 20, 2018

View reviewed changes

RobAtticus reviewed Apr 24, 2018

View reviewed changes

cevian force-pushed the plan_expand_hypertables branch from 931d7ca to 2787ecd Compare April 25, 2018 19:26

erimatnor requested changes Apr 26, 2018

View reviewed changes

cevian force-pushed the plan_expand_hypertables branch 2 times, most recently from 717fa69 to 856b2e5 Compare April 29, 2018 21:19

cevian force-pushed the plan_expand_hypertables branch 3 times, most recently from 7f00c74 to f0e23c2 Compare April 30, 2018 18:45

cevian mentioned this pull request Apr 30, 2018

change planner cost for (merge)append nodes #500

Closed

mfreed added this to the 0.10.0 milestone May 7, 2018

mfreed mentioned this pull request May 7, 2018

Move space-partition exclusion to planner. #471

Closed

RobAtticus approved these changes May 7, 2018

View reviewed changes

sspieser mentioned this pull request May 14, 2018

Performance issues when using 10,000s of chunks #515

Closed

erimatnor requested changes May 14, 2018

View reviewed changes

cevian force-pushed the plan_expand_hypertables branch from f0e23c2 to 67b28dd Compare May 16, 2018 20:45

cevian force-pushed the plan_expand_hypertables branch from 67b28dd to 7c611d0 Compare May 16, 2018 20:49

erimatnor requested changes May 18, 2018

View reviewed changes

cevian force-pushed the plan_expand_hypertables branch from 6e67bd1 to b014c87 Compare May 18, 2018 20:53

erimatnor approved these changes May 25, 2018

View reviewed changes

cevian added 2 commits May 25, 2018 11:18

Bump pg version to test to 9.6.6

72fdda2

We hit a bug in 9.6.5 fixed in 9.6.6 by commit 77cd0dc. Also changed extension is transitioning check to not palloc anything. This is more efficient and probably has slightly less side-effects on bugs like this.

cevian force-pushed the plan_expand_hypertables branch from b014c87 to 72fdda2 Compare May 25, 2018 15:19

cevian merged commit ad34d6f into master May 25, 2018

RobAtticus deleted the plan_expand_hypertables branch June 26, 2018 18:26


		chunk_scan_ctx_foreach_chunk(ctx, chunk_is_complete, 1);

		return (ctx->data == NIL ? NULL : linitial(ctx->data));

+              	}
+              }
+              bool

Optimize planning times when hypertables have many chunks #502

Optimize planning times when hypertables have many chunks #502

Conversation

cevian commented Apr 17, 2018

cevian commented Apr 17, 2018

RobAtticus left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

goodkiller commented Apr 25, 2018

erimatnor commented Apr 25, 2018

goodkiller commented Apr 25, 2018

RobAtticus commented Apr 25, 2018

erimatnor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cevian commented Apr 29, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erimatnor May 8, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erimatnor May 8, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cevian commented May 16, 2018

RobAtticus commented May 16, 2018

erimatnor left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erimatnor left a comment

Choose a reason for hiding this comment

erimatnor May 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erimatnor May 8, 2018 •

edited

Loading

erimatnor May 8, 2018 •

edited

Loading

erimatnor May 25, 2018 •

edited

Loading