Add postgres datastore type #5403

JamesChristie · 2018-08-27T18:32:48Z

This is work that aims to integrate changed submitted here and here. No changes have been made to the gx package hashes as the work it depends upon has not yet been accepted.

Stebalien · 2018-08-27T19:25:32Z

Thank you for the PR! Just a bit of a heads-up, we're a bit oversubscribed on quite a few fronts at the moment and the changes to sql-datastore are non-trivial so it may be a while before we get a chance to do a thorough review.

Quick things:

You can run gx release patch in the dependencies yourself to release new (non-canonical) versions yourself.
You can then update these dependencies here by running gx update $NewHash. Getting the tests to pass first will help speed up the review process significantly.

JamesChristie · 2018-08-28T13:21:19Z

I'm planning to do just that as soon as the PRs this depends on are merged! As soon as those are done, I'll update this pull; for the time being, it's here for posterity's sake.

magik6k · 2018-12-12T16:23:39Z

repo/fsrepo/datastores.go

+}
+
+type postgresDatastoreConfig struct {
+	path string


path isn't needed here either.

magik6k · 2018-12-12T16:37:29Z

repo/fsrepo/datastores.go

+}
+
+func (postgresDatastoreConfig) Create(path string) (repo.Datastore, error) {
+	pg := postgresdb.Options{}


This should get Host/User/Password/Database from config.

When initializing node with this backend user could pass those using env vars. in config profile we'd just call os.Getenv.

We then could provide a simple script like the one below to avoid storing passwords in shell history

#!/bin/sh [check args] export IPFS_PG_HOST=$1 export IPFS_PG_USER=$2 export IPFS_PG_DATABASE=$3 printf "${IPFS_PG_USER}@${IPFS_PG_HOST} password:" stty -echo read IPFS_PG_PASSWORD stty echo export IPFS_PG_PASSWORD ipfs init --profile=postgres

Thanks for the feedback! Making these changes now. I think we could also use the PGPASSFILE env variable to avoid directly storing the password as an env variable.

magik6k · 2018-12-12T16:42:05Z

repo/fsrepo/datastores.go

+func (c *postgresDatastoreConfig) DiskSpec() DiskSpec {
+	return map[string]interface{}{
+		"type": "postgresds",
+		"path": c.path,


path is not needed here, but I'm not sure what to put here that makes sense. My guess would be user+database, as host/password may change frequently in some setups.

magik6k

Sorry for taking so long and you having to fight with gx, but this is getting there.

Some things in other places:

Some of my pervious review comments still need addressing Implement go-ipfs datastore interface for postgres (#2) go-ds-sql#2
Git history in Add postgres profile go-ipfs-config#5 got a bit messy

There are also some merge conflicts here, if you don't want to fight with them, once other review comments are addressed I can fix them for you (it will likely involve gx too)

But other than that it's looking good.

.gx/lastpubver

.gitignore

plugin/plugins/postgresds/postgresds.go

scripts/ipfs_postgres.sh

i-norden · 2019-01-08T20:50:14Z

I believe I've addressed all of the above issues. The required sql-datastore PR has been merged but the go-ipfs-config PR still needs to be merged, iptb-plugins will then need to be gx-ified with the new go-ipfs-config hash, and then finally go-ipfs can be gx-ified with the new go-ipfs-config and iptb-plugins hashes.

magik6k

(1 nit + needs rebase), rest looks good

magik6k · 2019-01-11T16:32:06Z

plugin/loader/preload_list

@@ -8,3 +8,4 @@ ipldgit github.com/ipfs/go-ipfs/plugin/plugins/git 0
 badgerds github.com/ipfs/go-ipfs/plugin/plugins/badgerds 0
 flatfs github.com/ipfs/go-ipfs/plugin/plugins/flatfs 0
 levelds github.com/ipfs/go-ipfs/plugin/plugins/levelds 0
+postgresds github.com/ipfs/go-ipfs/plugin/plugins/postgresds 0


While it may be useful for quite a few people, I wouldn't preload this plugin by default (leaving commented out is fine)

Thank you for all the help, magik6k!

magik6k · 2019-01-12T10:38:49Z

Jenkins seems to be unable to fetch dependencies from you - https://ci.ipfs.team/blue/organizations/jenkins/IPFS%2Fgo-ipfs/detail/PR-5403/44/pipeline. (gitcop warning needs fixing tee)

i-norden · 2019-01-14T17:10:04Z

Jenkins seems to be unable to fetch dependencies from you - https://ci.ipfs.team/blue/organizations/jenkins/IPFS%2Fgo-ipfs/detail/PR-5403/44/pipeline. (gitcop warning needs fixing tee)

After making some very minor last minute changes to sql-datastore.. I forgot to re-gx-ify the package and so it was merged with the wrong hash :(

My apologies, sql-datastore will need a gx release patch performed.

i-norden · 2019-01-24T18:49:43Z

sql-datastore has been updated with the correct hash and make gx-deps is passing where it previous failed, but we still run into an issue with the sql-datastore dependency when running the shell script https://ci.ipfs.team/blue/organizations/jenkins/IPFS%2Fgo-ipfs/detail/PR-5403/51/pipeline.

I am also confused because the travis-ci builds are passing on OSX and only run into this issue with sql-datastore with the Linux builds.

Before this can be merged go-ipfs-config PR needs to be merged and its hash updated here and in iptb-plugins, and then the iptb-plugins hash needs to be updated here as well.

Stebalien · 2019-01-24T19:46:07Z

Lol. It looks like gx is case sensitive when it comes to language names.

Stebalien · 2019-01-24T19:49:02Z

Try again with: QmZmPKg1RKY2Cy5czKEBJTfDYzenbGww2pFNQKxLSKrPRB

(you're not the first to find this, apparently: whyrusleeping/gx#187)

i-norden · 2019-01-24T23:40:02Z

Yay that seems to have fixed the problem, thanks! It looks like the QmZmPKg1RKY2Cy5czKEBJTfDYzenbGww2pFNQKxLSKrPRB hash didn't capture the changes from ipfs/go-ds-sql#6 so we might want to gx release patch one more time although it appears the error we are getting now is just due to waiting on the updated go-ipfs-config dependencies.

License: MIT Signed-off-by: User Name <email@address>

…ies; undo gx releases; address feedback; gx hashes will need to be updated once go-ipfs-config PR is merged and gx released"

Stebalien · 2019-01-24T23:50:45Z

That was just a bug that has been fixed in master. I've rebased for you.

JamesChristie requested a review from Kubuxu as a code owner August 27, 2018 18:32

JamesChristie force-pushed the postgres branch 2 times, most recently from df6caaa to ce3bfc2 Compare August 27, 2018 18:36

Stebalien added the kind/feature A new feature label Aug 27, 2018

i-norden mentioned this pull request Dec 11, 2018

gx publish 1.0.7 ipfs/iptb-plugins#9

Closed

i-norden force-pushed the postgres branch from 3d9bef0 to 3c1e801 Compare December 11, 2018 19:34

magik6k reviewed Dec 12, 2018

View reviewed changes

i-norden mentioned this pull request Dec 12, 2018

update deps; add GetSize method; gx publish 1.0.0 vulcanize/sql-datastore#4

Merged

i-norden force-pushed the postgres branch from 91b4a67 to a0db370 Compare December 13, 2018 06:35

magik6k reviewed Dec 13, 2018

View reviewed changes

i-norden force-pushed the postgres branch 2 times, most recently from 9b54bce to 0fd2dd3 Compare December 18, 2018 20:50

magik6k self-requested a review December 18, 2018 21:17

i-norden force-pushed the postgres branch from 0fd2dd3 to 3e7ba5c Compare January 7, 2019 19:17

magik6k approved these changes Jan 11, 2019

View reviewed changes

i-norden force-pushed the postgres branch from 3e7ba5c to 12e3d10 Compare January 11, 2019 17:54

i-norden force-pushed the postgres branch 2 times, most recently from ba74810 to 1534b75 Compare January 24, 2019 18:31

i-norden force-pushed the postgres branch 2 times, most recently from 29e3193 to b09225e Compare January 24, 2019 21:29

Add postgres datastore type

0ffca94

License: MIT Signed-off-by: User Name <email@address>

move postgres to plugins; test in head context with updated dependenc…

d8f9c3b

…ies; undo gx releases; address feedback; gx hashes will need to be updated once go-ipfs-config PR is merged and gx released"

Stebalien force-pushed the postgres branch from b09225e to d8f9c3b Compare January 24, 2019 23:50

ghost assigned Stebalien Jan 24, 2019

ghost added the status/in-progress In progress label Jan 24, 2019

i-norden mentioned this pull request Jan 25, 2019

Add postgres profile ipfs/go-ipfs-config#5

Merged

momack2 added this to In Progress in ipfs/go-ipfs May 9, 2019

i-norden mentioned this pull request Jun 27, 2019

Seed node vulcanize/vulcanizedb#114

Closed

Stebalien removed their assignment Apr 22, 2021

guseggert mentioned this pull request Mar 4, 2022

feat: add grpcds support #8734

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add postgres datastore type #5403

Add postgres datastore type #5403

JamesChristie commented Aug 27, 2018 •

edited

Stebalien commented Aug 27, 2018

JamesChristie commented Aug 28, 2018

magik6k Dec 12, 2018

magik6k Dec 12, 2018

i-norden Dec 12, 2018

magik6k Dec 12, 2018

magik6k left a comment

i-norden commented Jan 8, 2019

magik6k left a comment •

edited

magik6k Jan 11, 2019

i-norden Jan 11, 2019

magik6k commented Jan 12, 2019

i-norden commented Jan 14, 2019

i-norden commented Jan 24, 2019

Stebalien commented Jan 24, 2019

Stebalien commented Jan 24, 2019

i-norden commented Jan 24, 2019 •

edited

Stebalien commented Jan 24, 2019

Add postgres datastore type #5403

Are you sure you want to change the base?

Add postgres datastore type #5403

Conversation

JamesChristie commented Aug 27, 2018 • edited

Stebalien commented Aug 27, 2018

JamesChristie commented Aug 28, 2018

magik6k Dec 12, 2018

Choose a reason for hiding this comment

magik6k Dec 12, 2018

Choose a reason for hiding this comment

i-norden Dec 12, 2018

Choose a reason for hiding this comment

magik6k Dec 12, 2018

Choose a reason for hiding this comment

magik6k left a comment

Choose a reason for hiding this comment

i-norden commented Jan 8, 2019

magik6k left a comment • edited

Choose a reason for hiding this comment

magik6k Jan 11, 2019

Choose a reason for hiding this comment

i-norden Jan 11, 2019

Choose a reason for hiding this comment

magik6k commented Jan 12, 2019

i-norden commented Jan 14, 2019

i-norden commented Jan 24, 2019

Stebalien commented Jan 24, 2019

Stebalien commented Jan 24, 2019

i-norden commented Jan 24, 2019 • edited

Stebalien commented Jan 24, 2019

JamesChristie commented Aug 27, 2018 •

edited

magik6k left a comment •

edited

i-norden commented Jan 24, 2019 •

edited