ADB-33 Add zstd compression algorithm for storage

README.md

@@ -110,9 +110,9 @@ building, see the README at the following repositories:
     2. conan remote add conan-gpdb https://api.bintray.com/conan/greenplum-db/gpdb-oss
     3. conan install --build
        * This command will fetch the orca and xerces artifacts from bintray repository, build and install them.
-       * The header and library files will be copied to the location specified by imports section of conanfile.txt in depends directory. 
+       * The header and library files will be copied to the location specified by imports section of conanfile.txt in depends directory.
        In case, the files should be copied elsewhere, please change the location.
-    
+
 ### Build the database
 
 ```
@@ -217,10 +217,20 @@ make distclean
 
 PXF is an extension framework for GPDB to enable fast access to external hadoop datasets.
 Refer to [PXF extension](https://github.com/greenplum-db/gpdb/tree/master/gpAux/extensions/pxf) for more information.
-Currently, GPDPB is built with PXF by default (--enable-pxf is on).
+Currently, GPDB is built with PXF by default (`--enable-pxf` is on).
 In order to build GPDB without pxf, simply invoke `./configure` with additional option `--disable-pxf`.
 PXF requires curl, so `--enable-pxf` is not compatible with the `--without-libcurl` option.
 
+### Building GPDB with Zstandard
+
+[Zstandard](https://github.com/facebook/zstd) is a fast real-time compression algorithm
+which offers a wide range of compression / speed trade-off. Zstandard is intended to become
+a default compression for most applications in GPDB. Currently, it is available as a
+compression option for append-only and append-only column-oriented tables.
+
+To build GPDB with Zstandard support, install packages `libzstd` and `libzstd-devel` and
+invoke `./configure` with an additional option `--with-zstd`.
+
 ### Building GPDB with gpperfmon enabled
 
 gpperfmon tracks a variety of queries, statistics, system properties, and metrics.
@@ -314,7 +324,7 @@ future releases.
 
 Greenplum is developed on GitHub, and anybody wishing to contribute to it will
 have to [have a GitHub account](https://github.com/signup/free) and be familiar
-with [Git tools and workflow](https://wiki.postgresql.org/wiki/Working_with_Git). 
+with [Git tools and workflow](https://wiki.postgresql.org/wiki/Working_with_Git).
 It is also recommend that you follow the [developer's mailing list](http://greenplum.org/#contribute)
 since some of the contributions may generate more detailed discussions there.
 
@@ -334,10 +344,10 @@ easier to submit one instead of claiming an "obvious fix" exception.
 
 If the contribution you're submitting is original work, you can assume that Pivotal
 will release it as part of an overall Greenplum release available to the downstream
-consumers under the Apache License, Version 2.0. However, in addition to that, Pivotal 
+consumers under the Apache License, Version 2.0. However, in addition to that, Pivotal
 may also decide to release it under a different license (such as [PostgreSQL License](https://www.postgresql.org/about/licence/) to the upstream consumers that require it. A typical example here would be Pivotal
 upstreaming your contribution back to PostgreSQL community (which can be done either
-verbatim or your contribution being upstreamed as part of the larger changeset). 
+verbatim or your contribution being upstreamed as part of the larger changeset).
 
 If the contribution you're submitting is NOT original work you have to indicate the name
 of the license and also make sure that it is similar in terms to the Apache License 2.0.
@@ -360,7 +370,7 @@ code. Even when your proposal gets validated by the community, we still recommen
 doing the actual work as a series of small, self-contained commits. This makes
 the reviewer's job much easier and increases the timeliness of feedback.
 
-When it comes to C and C++ parts of Greenplum, we try to follow 
+When it comes to C and C++ parts of Greenplum, we try to follow
 [PostgreSQL Coding Conventions](https://www.postgresql.org/docs/devel/static/source.html).
 In addition to that we require that:
    * All Python code passes [Pylint](https://www.pylint.org/)
@@ -370,11 +380,11 @@ We recommend using ```git diff --color``` when reviewing your changes so that yo
 don't have any spurious whitespace issues in the code that you submit.
 
 All new functionality that is contributed to Greenplum should be covered by regression
-tests that are contributed alongside it. If you are uncertain on how to test or document 
-your work, please raise the question on the gpdb-dev mailing list and the developer 
+tests that are contributed alongside it. If you are uncertain on how to test or document
+your work, please raise the question on the gpdb-dev mailing list and the developer
 community will do its best to help you.
 
-At the very minimum you should always be running 
+At the very minimum you should always be running
 ```make installcheck-world```
 to make sure that you're not breaking anything.
 
@@ -389,15 +399,15 @@ you can be sure whether your changes may need to be forward-ported.
 
 ### Submission timing
 
-To improve the odds of the right discussion of your patch or idea happening, pay attention 
-to what the community work cycle is. For example, if you send in a brand new idea in the 
+To improve the odds of the right discussion of your patch or idea happening, pay attention
+to what the community work cycle is. For example, if you send in a brand new idea in the
 beta phase of a release, we may defer review or target its inclusion for a later version.
 Feel free to ask on the mailing list to learn more about the Greenplum release policy and timing.
 
 ### Patch submission
 
 Once you are ready to share your work with the Greenplum core team and the rest of
-the Greenplum community, you should push all the commits to a branch in your own 
+the Greenplum community, you should push all the commits to a branch in your own
 repository forked from the official Greenplum and [send us a pull request](https://help.github.com/articles/about-pull-requests/).
 
 For now, we require all pull requests to be submitted against the main master
@@ -424,13 +434,13 @@ to a pull request in your email.
 
 ### Patch review
 
-A submitted pull request with passing validation checks is assumed to be available 
-for peer review. Peer review is the process that ensures that contributions to Greenplum 
+A submitted pull request with passing validation checks is assumed to be available
+for peer review. Peer review is the process that ensures that contributions to Greenplum
 are of high quality and align well with the road map and community expectations. Every
 member of the Greenplum community is encouraged to review pull requests and provide
 feedback. Since you don't have to be a core team member to be able to do that, we
 recommend following a stream of pull reviews to anybody who's interested in becoming
-a long-term contributor to Greenplum. As [Linus would say](https://en.wikipedia.org/wiki/Linus's_Law) 
+a long-term contributor to Greenplum. As [Linus would say](https://en.wikipedia.org/wiki/Linus's_Law)
 "given enough eyeballs, all bugs are shallow".
 
 One outcome of the peer review could be a consensus that you need to modify your
@@ -439,11 +449,11 @@ a branch from which a pull request was sent. Those additional commits will be th
 visible to all of the reviewers.
 
 A peer review converges when it receives at least one +1 and no -1s votes from
-the participants. At that point you should expect one of the core team 
+the participants. At that point you should expect one of the core team
 members to pull your changes into the project.
 
-Greenplum prides itself on being a collaborative, consensus-driven environment. 
-We do not believe in vetoes and any -1 vote casted as part of the peer review 
+Greenplum prides itself on being a collaborative, consensus-driven environment.
+We do not believe in vetoes and any -1 vote casted as part of the peer review
 has to have a detailed technical explanation of what's wrong with the change.
 Should a strong disagreement arise it may be advisable to take the matter onto
 the mailing list since it allows for a more natural flow of the conversation.

configure

@@ -713,6 +713,7 @@ with_apu_config
 with_apr_config
 with_libcurl
 with_rt
+with_zstd
 with_zlib
 with_system_tzdata
 with_libxslt
@@ -881,6 +882,7 @@ with_libxml
 with_libxslt
 with_system_tzdata
 with_zlib
+with_zstd
 with_rt
 with_libcurl
 with_apr_config
@@ -1576,6 +1578,7 @@ STRING
   --with-libxslt          use XSLT support when building contrib/xml2
   --with-system-tzdata=DIR  use system time zone data in DIR
   --without-zlib          do not use Zlib
+  --with-zstd             build with Zstandard support (requires zstd library)
   --without-rt            do not use Realtime Library
   --without-libcurl       do not use libcurl
   --with-apr-config=PATH  path to apr-1-config utility
@@ -7753,6 +7756,36 @@ fi
 
 
 
+#
+# zstd
+#
+
+pgac_args="$pgac_args with_zstd"
+
+
+# Check whether --with-zstd was given.
+if test "${with_zstd+set}" = set; then :
+  withval=$with_zstd;
+  case $withval in
+    yes)
+      :
+      ;;
+    no)
+      :
+      ;;
+    *)
+      as_fn_error $? "no argument expected for --with-zstd option" "$LINENO" 5
+      ;;
+  esac
+
+else
+  with_zstd=no
+
+fi
+
+
+
+
 #
 # Realtime library
 #
@@ -10120,6 +10153,56 @@ fi
 
 fi
 
+if test "$with_zstd" = yes; then
+  { $as_echo "$as_me:${as_lineno-$LINENO}: checking for ZSTD_compressCCtx in -lzstd" >&5
+$as_echo_n "checking for ZSTD_compressCCtx in -lzstd... " >&6; }
+if ${ac_cv_lib_zstd_ZSTD_compressCCtx+:} false; then :
+  $as_echo_n "(cached) " >&6
+else
+  ac_check_lib_save_LIBS=$LIBS
+LIBS="-lzstd  $LIBS"
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h.  */
+
+/* Override any GCC internal prototype to avoid an error.
+   Use char because int might match the return type of a GCC
+   builtin and then its argument prototype would still apply.  */
+#ifdef __cplusplus
+extern "C"
+#endif
+char ZSTD_compressCCtx ();
+int
+main ()
+{
+return ZSTD_compressCCtx ();
+  ;
+  return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+  ac_cv_lib_zstd_ZSTD_compressCCtx=yes
+else
+  ac_cv_lib_zstd_ZSTD_compressCCtx=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+    conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_zstd_ZSTD_compressCCtx" >&5
+$as_echo "$ac_cv_lib_zstd_ZSTD_compressCCtx" >&6; }
+if test "x$ac_cv_lib_zstd_ZSTD_compressCCtx" = xyes; then :
+  cat >>confdefs.h <<_ACEOF
+#define HAVE_LIBZSTD 1
+_ACEOF
+
+  LIBS="-lzstd $LIBS"
+
+else
+  as_fn_error $? "zstd library not found." "$LINENO" 5
+fi
+
+fi
+
 if test "$enable_spinlocks" = yes; then
 
 $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h
@@ -12455,6 +12538,18 @@ else
 fi
 
 
+fi
+
+# Check for zstd.h
+if test "$with_zstd" = yes; then
+  ac_fn_c_check_header_mongrel "$LINENO" "zstd.h" "ac_cv_header_zstd_h" "$ac_includes_default"
+if test "x$ac_cv_header_zstd_h" = xyes; then :
+
+else
+  as_fn_error $? "header file <zstd.h> is required for zstd support" "$LINENO" 5
+fi
+
+
 fi
 
 # On BSD, cpp test for net/if.h will fail unless sys/socket.h

configure.in

@@ -987,6 +987,13 @@ PGAC_ARG_BOOL(with, zlib, yes,
               [  --without-zlib          do not use Zlib])
 AC_SUBST(with_zlib)
 
+#
+# zstd
+#
+PGAC_ARG_BOOL(with, zstd, no,
+              [build with Zstandard support (requires zstd library)])
+AC_SUBST(with_zstd)
+
 #
 # Realtime library
 #
@@ -1175,6 +1182,11 @@ failure.  It is possible the compiler isn't looking in the proper directory.
 Use --without-zlib to disable zlib support.])])
 fi
 
+if test "$with_zstd" = yes; then
+  AC_CHECK_LIB(zstd, ZSTD_compressCCtx, [],
+               [AC_MSG_ERROR([zstd library not found.])])
+fi
+
 if test "$enable_spinlocks" = yes; then
   AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.])
 else
@@ -1331,6 +1343,11 @@ else
   AC_CHECK_HEADER(bzlib.h, [], [AC_MSG_ERROR([header file <bzlib.h> is required for bzip2 support])], [])
 fi
 
+# Check for zstd.h
+if test "$with_zstd" = yes; then
+  AC_CHECK_HEADER(zstd.h, [], [AC_MSG_ERROR([header file <zstd.h> is required for zstd support])])
+fi
+
 # On BSD, cpp test for net/if.h will fail unless sys/socket.h
 # is included first.
 AC_CHECK_HEADERS(net/if.h, [], [],