Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Support PGXN. (on top of PR-141) #150

Closed
wants to merge 2 commits into from

3 participants

@haradh1
Collaborator

Please look at only 86568d6, as this is atop of PR-141.

The requirement of PGXN support is, to let users do

 $ pgxn install madlib

to install extension files in postgres install directory.  After that,
users can CREATE EXTENSION in SQL.

The way pgxn client works is download the zip file, unpack it, enter
the directory, configure if any, make install.  To be compatible with
this, we create a Makefile in the top directory when configured, and
let "make install" be "make -C build extension-install".  This becomes
a little surprising behavior to the normal usage out of PGXN, so
the configure creates such Makefile only if the packager indicates,
by touching PGXN_ZIP file in the top directory.  This is automated
with deploy/pgxn.sh, which clones the latest git and put an empty
PGXN_ZIP file, then zip it.

This commit also contains META.json, which is the only requirement
to upload a module to PGXN distribution site.
haradh1 added some commits
@haradh1 haradh1 Support CREATE EXTENSION, take two.
Instead of the previous attempt to support this in madpack time, this time
we try to make the build system to create EXTENSION files so that upcoming
PGXN support can handle it gracefully.  For users who want to install it
as an extension, the process is to download source, configure, and
"make extension-install."  Still the extension is not relocatable, users
can specify schema name on CREATE EXTENSION with SCHEMA option.
3356596
@haradh1 haradh1 Support PGXN.
The requirement of PGXN support is, to let users do

 $ pgxn install madlib

to install extension files in postgres install directory.  After that,
users can CREATE EXTENSION in SQL.

The way pgxn client works is download the zip file, unpack it, enter
the directory, configure if any, make install.  To be compatible with
this, we create a Makefile in the top directory when configured, and
let "make install" be "make -C build extension-install".  This becomes
a little surprising behavior to the normal usage out of PGXN, so
the configure creates such Makefile only if the packager indicates,
by touching PGXN_ZIP file in the top directory.  This is automated
with deploy/pgxn.sh, which clones the latest git and put an empty
PGXN_ZIP file, then zip it.

This commit also contains META.json, which is the only requirement
to upload a module to PGXN distribution site.
86568d6
@cwelton cwelton commented on the diff
META.json
@@ -0,0 +1,37 @@
+{
+ "name": "madlib",
+ "abstract": "An open source in-detabase analytics library",
+ "description": "An open source in-database analytics library",
+ "version": "0.3.0-alpha",
@cwelton Owner
cwelton added a note

Will we need to maintain this version separate from src/config/Version.yml, any way we can have this automatic so that we don't have to maintain the version number in multiple places?

@haradh1 Collaborator
haradh1 added a note

First of all, this string is apart from madlib version, strictly speaking. PGXN distinguishes the version string of "distribution" (the packaged zip file) and the actual module extension, which is described below "provide" section.

And I really want to automate the version string with Version.yml. It may be possible with configure, but this file should be located in the top directory which is out of build/, and may make packaging process complicated.

@fschopp Collaborator
fschopp added a note

Couldn't the META.json file be generated if configure is called by pgxn? Or is META.json necessary in the zip file? In that case, it might only be necessary when deployed to pgxn?

Bottom line: I am wondering if we can get rid of this file in the "normal" repository.

@haradh1 Collaborator
haradh1 added a note

Yes, META.json is necessary in the zip file. I think we could automate PGXN deployment including this file, but the "how" is still not clear.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@cwelton
Owner

We should be sure to include some sort of automated test around pgxn install support for Postgres. Hopefully our firewalls allow for that...

@haradh1
Collaborator

For now the http access to pgxn including its API is ok. The client only uses http. But not sure if in some days it's banned :(

@haradh1
Collaborator

We need an account in http://manager.pgxn.org/ to upload finally.

@fschopp fschopp commented on the diff
src/madpack/sort-module.py
((7 lines not shown))
+string contains resolving the dependencies. Example:
+
+ $ sort-module.py a/regress/b a/svec/b a/array_ops/b
+ > a/svec/b a/regress/b a/array_opts/b
+
+Note this script assumes to be run at the exact current directory,
+so you need change directory first to run it.
+"""
+
+import configyml
+
+
+portspecs = configyml.get_modules("../config")
+
+def find_order(path):
+ for i, moduleinfo in enumerate(portspecs['modules']):
@fschopp Collaborator
fschopp added a note

Minor issue, but all our Python code uses 4 spaces for indentation.

@haradh1 Collaborator
haradh1 added a note

That's a big issue, indeed. I'll fix it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@fschopp fschopp commented on the diff
deploy/pgxn.sh
@@ -0,0 +1,12 @@
@fschopp Collaborator
fschopp added a note

Why don't you you clone the current repository instead of the remote one? Also, I wonder if we'd rather have a CMake script instead of a shell script, like for the other deployment targets.

@haradh1 Collaborator
haradh1 added a note

Hmm, that's a good idea to clone the current repository.

@haradh1 Collaborator
haradh1 added a note

Do you mean RPM package by "the other deployment targets"? The difference from it is that PGXN package needs almost raw source tree. I'm not sure about "install" process in cmake, but as I created "extension-install" target for EXTENSION, the cmake install process is not applicable here, I think.

@fschopp Collaborator
fschopp added a note

I thought the product of the PGXN deployment target would be a zip file. Sure, there are some differences in what it includes, but still the expected behavior would be the same as for other deployment targets like RPM or a Mac OS X package installer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@fschopp fschopp commented on the diff
META.json
@@ -0,0 +1,37 @@
+{
+ "name": "madlib",
+ "abstract": "An open source in-detabase analytics library",
+ "description": "An open source in-database analytics library",
@fschopp Collaborator
fschopp added a note

s/open source/open-source/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@fschopp
Collaborator

Looks fine for now. However, it would be nice if we had a few comments in the repository. Can you add the description of this pull request to a prominent location within one of the pgxn support files?

@haradh1
Collaborator

Thanks for your inputs. Pushed to master.

@haradh1 haradh1 closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on May 7, 2012
  1. @haradh1

    Support CREATE EXTENSION, take two.

    haradh1 authored
    Instead of the previous attempt to support this in madpack time, this time
    we try to make the build system to create EXTENSION files so that upcoming
    PGXN support can handle it gracefully.  For users who want to install it
    as an extension, the process is to download source, configure, and
    "make extension-install."  Still the extension is not relocatable, users
    can specify schema name on CREATE EXTENSION with SCHEMA option.
Commits on May 10, 2012
  1. @haradh1

    Support PGXN.

    haradh1 authored
    The requirement of PGXN support is, to let users do
    
     $ pgxn install madlib
    
    to install extension files in postgres install directory.  After that,
    users can CREATE EXTENSION in SQL.
    
    The way pgxn client works is download the zip file, unpack it, enter
    the directory, configure if any, make install.  To be compatible with
    this, we create a Makefile in the top directory when configured, and
    let "make install" be "make -C build extension-install".  This becomes
    a little surprising behavior to the normal usage out of PGXN, so
    the configure creates such Makefile only if the packager indicates,
    by touching PGXN_ZIP file in the top directory.  This is automated
    with deploy/pgxn.sh, which clones the latest git and put an empty
    PGXN_ZIP file, then zip it.
    
    This commit also contains META.json, which is the only requirement
    to upload a module to PGXN distribution site.
This page is out of date. Refresh to see the latest.
View
37 META.json
@@ -0,0 +1,37 @@
+{
+ "name": "madlib",
+ "abstract": "An open source in-detabase analytics library",
+ "description": "An open source in-database analytics library",
@fschopp Collaborator
fschopp added a note

s/open source/open-source/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ "version": "0.3.0-alpha",
@cwelton Owner
cwelton added a note

Will we need to maintain this version separate from src/config/Version.yml, any way we can have this automatic so that we don't have to maintain the version number in multiple places?

@haradh1 Collaborator
haradh1 added a note

First of all, this string is apart from madlib version, strictly speaking. PGXN distinguishes the version string of "distribution" (the packaged zip file) and the actual module extension, which is described below "provide" section.

And I really want to automate the version string with Version.yml. It may be possible with configure, but this file should be located in the top directory which is out of build/, and may make packaging process complicated.

@fschopp Collaborator
fschopp added a note

Couldn't the META.json file be generated if configure is called by pgxn? Or is META.json necessary in the zip file? In that case, it might only be necessary when deployed to pgxn?

Bottom line: I am wondering if we can get rid of this file in the "normal" repository.

@haradh1 Collaborator
haradh1 added a note

Yes, META.json is necessary in the zip file. I think we could automate PGXN deployment including this file, but the "how" is still not clear.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ "maintainer": "MADlib development team",
+ "license": "bsd",
+ "provides": {
+ "madlib": {
+ "file": "madlib--0.3.0.sql"
+ "docfile": "ReadMe.txt",
+ "version": "0.3.0"
+ }
+ },
+ "resources": {
+ "homepage": "http://madlib.net/",
+ "bugtracker": {
+ "web": "http://jira.madlib.net/"
+ },
+ "repository": {
+ "url": "https://github.com/madlib/madlib.git"
+ "web": "https://github.com/madlib/madlib",
+ "type": "git"
+ }
+ },
+ "release_status": "testing",
+ "tags": [
+ "machine learning",
+ "analytics"
+ ],
+
+
+ "meta-spec": {
+ "version": "1.0.0",
+ "url": "http://pgxn.org/meta/spec.txt"
+ }
+}
View
21 configure
@@ -17,3 +17,24 @@ mkdir -p build
cd build
rm -f CMakeCache.txt
cmake "$@" ..
+
+cd ..
+
+cat <<MAKEFILE >Makefile
+
+all %:
+ \$(MAKE) -C build \$@
+clean:
+ rm -f Makefile
+ rm -rf build
+MAKEFILE
+
+if [ -e PGXN_ZIP ]; then
+ cat <<MAKEFILE >>Makefile
+install: extension-install
+# keep a tab to do nothing for install
+
+extension-install:
+ \$(MAKE) -C build \$@
+MAKEFILE
+fi
View
12 deploy/pgxn.sh
@@ -0,0 +1,12 @@
+#!/bin/sh
+
+# This is the version string for the distribution
+VERSION=0.3.0
+TEMPDIR=`mktemp -d -t madlib`
+ORIG=`pwd`
+cd $TEMPDIR
+git clone git://github.com/madlib/madlib.git madlib-$VERSION
+touch madlib-$VERSION/PGXN_ZIP
+zip -x "madlib-$VERSION/.git/*" -r $ORIG/madlib.zip ./madlib-$VERSION
+rm -rf madlib-$VERSION
+
View
34 src/madpack/sort-module.py
@@ -0,0 +1,34 @@
+#!/usr/bin/python
+
+"""
+sort-module.py
+
+Given number of strings, sort them based on the module name which the
+string contains resolving the dependencies. Example:
+
+ $ sort-module.py a/regress/b a/svec/b a/array_ops/b
+ > a/svec/b a/regress/b a/array_opts/b
+
+Note this script assumes to be run at the exact current directory,
+so you need change directory first to run it.
+"""
+
+import configyml
+
+
+portspecs = configyml.get_modules("../config")
+
+def find_order(path):
+ for i, moduleinfo in enumerate(portspecs['modules']):
@fschopp Collaborator
fschopp added a note

Minor issue, but all our Python code uses 4 spaces for indentation.

@haradh1 Collaborator
haradh1 added a note

That's a big issue, indeed. I'll fix it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ modname = moduleinfo['name']
+ if modname in path:
+ return i
+ # return as the last if not found.
+ return len(portspecs['modules'])
+
+def main(args):
+ print " ".join(sorted(args, key = find_order))
+
+if __name__ == '__main__':
+ import sys
+ main(sys.argv[1:])
View
1  src/ports/postgres/9.1/CMakeLists.txt
@@ -1 +1,2 @@
add_current_postgresql_version()
+add_extension_support()
View
94 src/ports/postgres/CMakeLists.txt
@@ -236,6 +236,99 @@ macro(add_current_${PORT_LC}_version)
endif(${PORT_UC}_${_VERSION_UNDERSCORES}_FOUND)
endmacro(add_current_${PORT_LC}_version)
+# -- Extension support
+# If the DBMS can support CREATE EXTENSION, call this function
+# after add_current_PORT_version()
+function(add_extension_support)
+ get_filename_component(_VERSION "${CMAKE_CURRENT_SOURCE_DIR}" NAME)
+ string(REPLACE "." "_" PORT_VERSION_UNDERSCORE "${_VERSION}")
+ set(DBMS "${PORT_LC}_${PORT_VERSION_UNDERSCORE}")
+ set(DBMS_UC "${PORT_UC}_${PORT_VERSION_UNDERSCORE}")
+
+ foreach(CURRENT_FILE ${SQL_TARGET_FILES})
+ if(NOT ${CURRENT_FILE} MATCHES "/test/")
+ set(OUTFILE ${CURRENT_FILE})
+ string(REGEX REPLACE ".sql_in\$" ".sql" OUTFILE ${OUTFILE})
+ get_dir_name(OUTDIR ${OUTFILE})
+ get_filename_component(MODULE_NAME ${OUTDIR} NAME)
+
+ # This should match with madpack
+ # We use @extschema@ for its schema and let users
+ # decide the schema name when CREATE EXTENSION.
+ # Also, we install libmadlib.so into $libdir
+ # so madlib.control specifies MODULE_PATHNAME and we don't
+ # do it here.
+ set(M4_ARGS
+ "-P"
+ "-DMADLIB_SCHEMA=@extschema@"
+ "-DPLPYTHON_LIBDIR=${${DBMS_UC}_SHARE_DIR}/madlib/modules"
+ "-DMODULE_NAME=${MODULE_NAME}"
+ "-D${PORT_UC}"
+ "-I${CMAKE_CURRENT_BINARY_DIR}/madpack"
+ )
+ add_custom_command(OUTPUT "${OUTFILE}"
+ COMMAND ${CMAKE_COMMAND} -E make_directory "${OUTDIR}"
+ COMMAND ${M4_BINARY} ${M4_ARGS} "${CURRENT_FILE}" > "${OUTFILE}"
+ DEPENDS sqlFiles_${PORT_LC}
+ COMMENT "Preprocessing ${CURRENT_FILE} with m4."
+ )
+ list(APPEND MODULE_SQL_POST_FILES ${OUTFILE})
+ endif(NOT ${CURRENT_FILE} MATCHES "/test/")
+ endforeach(CURRENT_FILE)
+
+ configure_file("${PORT_SOURCE_DIR}/extension/madlib.control_in"
+ "${CMAKE_CURRENT_BINARY_DIR}/extension/madlib.control"
+ @ONLY
+ )
+ # base file contains miscellaneous operations such like creating history table
+ set(EXTENSION_SQL_BASE "${CMAKE_CURRENT_BINARY_DIR}/extension/madlib.sql.base")
+ set(extschema "@extschema@")
+ configure_file("${PORT_SOURCE_DIR}/extension/madlib.sql.base_in"
+ "${EXTENSION_SQL_BASE}"
+ @ONLY
+ )
+ unset(extschema)
+
+ # We use X.Y.Z style for version string for it is recommended in PGXN.
+ set(EXTENSION_SQL "${CMAKE_CURRENT_BINARY_DIR}/extension/madlib--${MADLIB_VERSION_MAJOR}.${MADLIB_VERSION_MINOR}.${MADLIB_VERSION_PATCH}.sql")
+
+ # The workhorse for extension target
+ # It is hard to interpret Modules.yml in this cmake file, thus we use
+ # sort-module.py to sort modules based on the dependencies.
+ add_custom_command(OUTPUT ${EXTENSION_SQL}
+ COMMAND cat ${EXTENSION_SQL_BASE}
+ `cd ${CMAKE_BINARY_DIR}/src/madpack &&
+ python sort-module.py ${MODULE_SQL_POST_FILES}` >${EXTENSION_SQL}
+ DEPENDS ${MODULE_SQL_POST_FILES} madpackFiles configFiles pythonFiles_${DBMS} madlib_${DBMS}
+ COMMENT "Concatenating into ${EXTENSION_SQL}"
+ )
+ # default target for extension
+ add_custom_target(extension
+ DEPENDS ${EXTENSION_SQL}
+ )
+
+ # clean target for extension
+ add_custom_target(extension-clean
+ COMMAND ${CMAKE_COMMAND} -E remove ${EXTENSION_SQL}
+ )
+
+ # install target for extension
+ # Unfortunately, cmake install command cannot be used here, as
+ # DIRECTORY install doesn't have custom target option.
+ add_custom_target(extension-install
+ COMMAND ${CMAKE_COMMAND} -E make_directory ${${DBMS_UC}_SHARE_DIR}/madlib
+ COMMAND ${CMAKE_COMMAND} -E copy_directory
+ ${CMAKE_CURRENT_BINARY_DIR}/modules ${${DBMS_UC}_SHARE_DIR}/madlib/modules
+ COMMAND ${CMAKE_COMMAND} -E copy
+ ${CMAKE_CURRENT_BINARY_DIR}/lib/libmadlib.so ${${DBMS_UC}_LIB_DIR}
+ COMMAND ${CMAKE_COMMAND} -E copy
+ ${EXTENSION_SQL} ${${DBMS_UC}_SHARE_DIR}/extension/
+ COMMAND ${CMAKE_COMMAND} -E copy
+ ${CMAKE_CURRENT_BINARY_DIR}/extension/madlib.control
+ ${${DBMS_UC}_SHARE_DIR}/extension/
+ DEPENDS ${EXTENSION_SQL}
+ )
+endfunction(add_extension_support)
# -- 6. Build shared library and copy version-specific file for all
# ${PORT_UC}_X_Y_PG_CONFIG macros defined by the user. If none has been
@@ -245,3 +338,4 @@ determine_target_versions(VERSIONS)
foreach(VERSION ${VERSIONS})
add_subdirectory(${VERSION})
endforeach(VERSION)
+
View
8 src/ports/postgres/cmake/FindPostgreSQL.cmake
@@ -21,6 +21,7 @@
# (see below)
# ${PKG_NAME}_FOUND - set to true if headers and binary were found
# ${PKG_NAME}_LIB_DIR - PostgreSQL library directory
+# ${PKG_NAME}_SHARE_DIR - PostgreSQL share directory
# ${PKG_NAME}_CLIENT_INCLUDE_DIR - client include directory
# ${PKG_NAME}_SERVER_INCLUDE_DIR - server include directory
# ${PKG_NAME}_EXECUTABLE - path to postgres binary
@@ -155,7 +156,12 @@ if(${PKG_NAME}_PG_CONFIG AND ${PKG_NAME}_CLIENT_INCLUDE_DIR)
OUTPUT_VARIABLE ${PKG_NAME}_LIB_DIR
OUTPUT_STRIP_TRAILING_WHITESPACE
)
-
+
+ execute_process(COMMAND ${${PKG_NAME}_PG_CONFIG} --sharedir
+ OUTPUT_VARIABLE ${PKG_NAME}_SHARE_DIR
+ OUTPUT_STRIP_TRAILING_WHITESPACE
+ )
+
architecture("${${PKG_NAME}_EXECUTABLE}" ${PKG_NAME}_ARCHITECTURE)
else()
message(STATUS "Found pg_config at \"${${PKG_NAME}_PG_CONFIG}\", "
View
5 src/ports/postgres/extension/madlib.control_in
@@ -0,0 +1,5 @@
+default_version = '@MADLIB_VERSION_MAJOR@.@MADLIB_VERSION_MINOR@.@MADLIB_VERSION_PATCH@'
+comment = 'A scalable in-database analytics library'
+relocatable = false
+module_pathname = '$libdir/libmadlib.so'
+requires = 'plpythonu'
View
5 src/ports/postgres/extension/madlib.sql.base_in
@@ -0,0 +1,5 @@
+CREATE TABLE @extschema@.migrationhistory(
+ id serial, version varchar(255), applied timestamp default current_timestamp
+);
+INSERT INTO @extschema@.migrationhistory(version)
+ VALUES('@MADLIB_VERSION_STRING@'::text);
Something went wrong with that request. Please try again.