v3.2.0 #229

eeeebbbbrrrr · 2017-10-23T20:11:22Z

A pull request for what will soon become v3.2.0. Release notes coming soon...

…low.

…pdated/deleted rows. This greatly simplifies everything, however there's still some concurrency problem lurking...

…vacuum.

…borted until the transaction actually commits, then they're deleted. This drastically reduces the number of docs we need to track in ES, and eliminates the "hints" type entirely, which is also great.

…Lion. It's not a usable library (maybe it is? I haven't actually tried), but being able to find compilation problems within CLion is nice

in elasticsearch.c#elasticsearch_vacuumCleanup() that causes an xid to be deleted that is otherwise still visible by a running transaction. Still looking into this. Also cleanup how we refresh a bit -- do a number of fewer refreshes now

important to account for the fact that they may represent xmin and/or xmax values. And we need to delete docs accordingly.

"data" and "xmax" along with the blocknumber as an integer to "data". VisibilityQueryHelper still has some debugging output that needs to be deleted.

…binary" to "data", and rename "_zdb_quick_lookup" in "xmax" to the same

found got deleted before we searched for it in "data"

…twice.

we put things into the database.

… part of ES 1.7.5. Also get rid of the "known_visible" check -- it's just extra cycles and confusing and unnecessary.

that xmax doc was created because of an "U"pdate or "D"elete. We don't use this for anything, but can be useful for debugging purposes.

Version of the "xmax" doc we want to delete. Additionally go ahead and allow the transaction id to be deleted from "aborted" since it's known-to-be-considered-aborted-by-all-current-and-future-transactions inside Postgres. Also add some comments about what's going on and of course, call the method from zdbam.c#vacuumcleanup

that we think are invisible to us, rather than running through every possible ItemPointer

eeeebbbbrrrr · 2017-10-23T20:47:53Z

elasticsearch/src/main/java/com/tcdi/zombodb/postgres/PostgresAggregationAction.java

@@ -54,7 +54,7 @@ protected void handleRequest(RestRequest request, RestChannel channel, Client cl
            SearchRequestBuilder builder = new SearchRequestBuilder(client);
            String input = request.content().toUtf8();
            final QueryRewriter rewriter = QueryRewriter.Factory.create(client, request.param("index"), request.param("preference"), input, true, true, true);
-            QueryBuilder qb = rewriter.rewriteQuery(true);
+            QueryBuilder qb = rewriter.rewriteQuery(false);


Don't need this boolean argument to QueryBuilder.rewriteQuery() anymore

eeeebbbbrrrr · 2017-10-23T20:49:44Z

elasticsearch/src/main/java/com/tcdi/zombodb/postgres/PostgresCountAction.java

@@ -47,7 +48,7 @@ protected void handleRequest(RestRequest request, RestChannel channel, Client cl
            BytesRestResponse response;
            QueryAndIndexPair query;

-            query = PostgresTIDResponseAction.buildJsonQueryFromRequestContent(client, request, !isSelectivityQuery, true, true, true);
+            query = PostgresTIDResponseAction.buildJsonQueryFromRequestContent(client, request, !isSelectivityQuery, true, true, false);


Don't need last boolean argument to PostgresTIDResponseAction.buildJsonQueryFromRequestContent()

eeeebbbbrrrr · 2017-10-23T20:51:00Z

elasticsearch/src/main/java/com/tcdi/zombodb/postgres/ZombodbBulkAction.java

+            Number cmin = (Number) data.get("_cmin");
+            Number sequence = (Number) data.get("_zdb_seq");    // -1 means an index build (CREATE INDEX)
+
+            {


TODO: In a future release, move this tuple encoding to the Postgres side of things so as to avoid manipulating and copying large byte arrays for every row we're indexing.

eeeebbbbrrrr · 2017-10-23T20:52:27Z

elasticsearch/src/main/java/com/tcdi/zombodb/postgres/ZombodbGetXidVacuumCandidatesAction.java

+
+        client.admin().indices().refresh(new RefreshRequestBuilder(client.admin().indices()).setIndices(index).request()).actionGet();
+
+        SearchRequestBuilder search = new SearchRequestBuilder(client)


document what we consider a possible vacuum-able transaction id, and why

eeeebbbbrrrr · 2017-10-23T20:54:07Z

elasticsearch/src/main/java/com/tcdi/zombodb/postgres/ZombodbVacuumSupportAction.java

+
+        client.admin().indices().refresh(new RefreshRequestBuilder(client.admin().indices()).setIndices(index).request()).actionGet();
+
+        SearchRequestBuilder search = new SearchRequestBuilder(client)


document what we're querying for here, and why.

Answer: It's every doc we think is currently invisible to us.

eeeebbbbrrrr · 2017-10-23T20:55:03Z

elasticsearch/src/main/java/com/tcdi/zombodb/postgres/ZombodbVacuumSupportAction.java

+        for (int i = 0; i < tmp.length; i++)
+            active[i] = Long.valueOf(tmp[i]);
+
+        client.admin().indices().refresh(new RefreshRequestBuilder(client.admin().indices()).setIndices(index).request()).actionGet();


document why we need to refresh before we vacuum.

Answer: so that we don't miss any invisible tuples from aborted transactions when zombodb.batch_mode = on that Postgres' VACUUM process thinks need to be vacuumed

eeeebbbbrrrr · 2017-10-23T20:59:06Z

postgres/src/test/sql/sharded-updates-and-deletes.sql

@@ -0,0 +1,36 @@
+DROP TABLE IF EXISTS suad;


delete this entire file -- didn't mean to ever commit it

eeeebbbbrrrr · 2017-10-23T21:02:23Z

Attached are artifacts for this Pull Request.

This release is going to require that indexes be re-created from scratch :(, so the process should be something like:

DROP INDEX ... for every ZDB index in every database
upgrade all nodes in ES cluster and restart
install new Postgres extension
ALTER EXTENSION zombodb UPDATE; for every database
VACUUM table for every table where you intend to create a ZDB index
re-CREATE INDEX ... for every ZDB index you dropped above

artifacts.zip

eeeebbbbrrrr · 2017-10-23T21:57:47Z

elasticsearch/src/main/java/com/tcdi/zombodb/query/VisibilityQueryHelper.java


 final class VisibilityQueryHelper {

-    private static final ConcurrentSkipListSet<Long> KNOWN_COMMITTED_XIDS = new ConcurrentSkipListSet<>();
+    private static class HeapTuple implements Comparable<HeapTuple> {


Pull this out into a standalone class and put some unit tests around its .equals() and .compareTo() methods as they were a source of bugs during development

be easily unit tested

ItemPointers for every possible (BlockNumber, OffsetNumber) in the table, using Postgres' visibility_map to short-circuit blocks when we can. This fixes an issue with not enough docs being deleted during (auto)VACUUM when zombodb.batch_mode = on.

eeeebbbbrrrr · 2017-10-25T07:47:40Z

Updated artifacts that fix a bug with (auto)vacuum and zombodb.batch_mode = on.

artifacts.zip

when deleting "xmax" docs. This is actually a perfectly normal and expected response in cases where we try to delete an "xmax" doc that's been replaced by a new doc between when we thought we needed to delete it and when we actually do. This really should have been part of 34377b9 Also, change logging message so that a tail postgresql.log | grep "zombodb vacuum" will actually find this error.

it as an error anyways.

eeeebbbbrrrr · 2017-10-26T04:54:55Z

Updated artifacts that fix the "VersionConflictEngineException" during (auto)vacuum.
artifacts.zip

eeeebbbbrrrr added 28 commits October 23, 2017 14:03

buildsystem support for Ubuntu Xenial

ef530d9

working on improving performance after v3.1.15 make things terribly s…

ceb9b77

…low.

turn on query cache where we can

7878d93

cleanup

603d3fe

rewrite recent work to instead just track updated "xmax" values for u…

7d87a46

…pdated/deleted rows. This greatly simplifies everything, however there's still some concurrency problem lurking...

remove debugging

e7306a4

fixing (concurrent) visibility issues with normal searching and with …

7f70fe0

…vacuum.

fix syntax error

1b850c4

move to tracking active transaction ids only. They're assumed to be a…

dd93ce7

…borted until the transaction actually commits, then they're deleted. This drastically reduces the number of docs we need to track in ES, and eliminates the "hints" type entirely, which is also great.

code cleanup

54651b5

compile to a library so that compilation actually works from within C…

8bba068

…Lion. It's not a usable library (maybe it is? I haven't actually tried), but being able to find compilation problems within CLion is nice

fix compilation problem on <C99

c87f29e

fix concurrency bug. When looking at aborted transaction ids, it's

03095f2

important to account for the fact that they may represent xmin and/or xmax values. And we need to delete docs accordingly.

only add the "aborted" xid once per transaction, during updates.

bb72e28

optimize VisibilityQueryHelper. Added some type="binary" fields to both

fbccc48

"data" and "xmax" along with the blocknumber as an integer to "data". VisibilityQueryHelper still has some debugging output that needs to be deleted.

more optimization cleanup. add a "_zdb_encoded_tuple" field of type "…

5e8404d

…binary" to "data", and rename "_zdb_quick_lookup" in "xmax" to the same

turn on query cache again.

e09b4fa

fix ArrayIndexOutOfBoundsException when the "data" for an "xmax" we

fc7152b

found got deleted before we searched for it in "data"

one last round of optimizations -- avoid looping over ctids we found …

068367e

…twice.

fixing two race conditions and do some cleanup around the order in which

8030676

we put things into the database.

switch from IntSet to IntOpenHashSet as the former doesn't seem to be…

e1c9351

… part of ES 1.7.5. Also get rid of the "known_visible" check -- it's just extra cycles and confusing and unnecessary.

tidy up the auto-generated mapping for "data" a little bit

c7c7781

add a field named "_zdb_reason" to the "xmax" type where we track if

a7d70d0

that xmax doc was created because of an "U"pdate or "D"elete. We don't use this for anything, but can be useful for debugging purposes.

cleanup vacuum process so we're only asking to vacuum away the tuples

04a1833

that we think are invisible to us, rather than running through every possible ItemPointer

delete debugging output

b87de25

bump version to v3.2.0

fe3c510

eeeebbbbrrrr commented Oct 23, 2017

View reviewed changes

fix upgrade script

d230c43

eeeebbbbrrrr commented Oct 23, 2017

View reviewed changes

eeeebbbbrrrr added 2 commits October 23, 2017 15:11

code review cleanup

c5e1bf9

follow-up commit to code review

565e9bc

eeeebbbbrrrr changed the base branch from master to develop October 23, 2017 21:20

eeeebbbbrrrr commented Oct 23, 2017

View reviewed changes

eeeebbbbrrrr added 4 commits October 23, 2017 15:59

pull HeapTuple out into a standalone class so it can

f3da592

be easily unit tested

cleanup and comments

dab1470

copyright headers on new files

245f700

eeeebbbbrrrr added 2 commits October 25, 2017 22:25

cleanup output when we suppress an error so that Postgres doesn't detect

db276cc

it as an error anyways.

eeeebbbbrrrr merged commit 9cf4dce into develop Oct 26, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v3.2.0 #229

v3.2.0 #229

eeeebbbbrrrr commented Oct 23, 2017

eeeebbbbrrrr Oct 23, 2017

eeeebbbbrrrr Oct 23, 2017

eeeebbbbrrrr Oct 23, 2017

eeeebbbbrrrr Oct 23, 2017

eeeebbbbrrrr Oct 23, 2017

eeeebbbbrrrr Oct 23, 2017

eeeebbbbrrrr Oct 23, 2017

eeeebbbbrrrr commented Oct 23, 2017 •

edited

Loading

eeeebbbbrrrr Oct 23, 2017

eeeebbbbrrrr commented Oct 25, 2017

eeeebbbbrrrr commented Oct 26, 2017


		client.admin().indices().refresh(new RefreshRequestBuilder(client.admin().indices()).setIndices(index).request()).actionGet();

		SearchRequestBuilder search = new SearchRequestBuilder(client)

v3.2.0 #229

v3.2.0 #229

Conversation

eeeebbbbrrrr commented Oct 23, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eeeebbbbrrrr commented Oct 23, 2017 • edited Loading

Choose a reason for hiding this comment

eeeebbbbrrrr commented Oct 25, 2017

eeeebbbbrrrr commented Oct 26, 2017

eeeebbbbrrrr commented Oct 23, 2017 •

edited

Loading