fsrepo migrations #950

jbenet · 2015-03-20T14:03:54Z

This PR adds migrations for fsrepo.

a very simple migration structure
a 0-to-1 migration adding the version file
a 1-to-2 migration moving things around to look more
like https://github.com/ipfs/specs/tree/master/repo/fs-repo

The 1-to-2 isn't done yet and will be the meat of this PR. Need

flat repo in blocks/
move repo lock (not needed in the migration, just code)
make api file (not needed in the migration, just code)
make hooks dir
incorporate .ipfs change (config: change default config dir name to .ipfs #975)

jbenet · 2015-03-20T19:23:02Z

@whyrusleeping not sure how you want to store things in keyspec, but is the layout in https://github.com/ipfs/specs/tree/master/repo/fs-repo#keys work for you? note the .{pub, pri, sym} endings.

whyrusleeping · 2015-03-21T01:21:46Z

@jbenet thats a little non-standard (i was going for ssh-compliant naming and formats) but I can work with it.

jbenet · 2015-04-01T23:08:31Z

see discussion in #975 (comment)

whyrusleeping · 2015-04-01T23:43:33Z

@jbenet why do you have a binary commited?

jbenet · 2015-04-01T23:45:59Z

@whyrusleeping because i fail. thanks

remove 0-to-1 binary

whyrusleeping · 2015-04-01T23:51:40Z

lets make sure it gets GCed out! 2MB makes a difference to new clones

whyrusleeping · 2015-04-02T01:10:43Z

this looks good to me, i like the approach.

whyrusleeping · 2015-04-07T06:03:16Z

@jbenet should the daemon check the repo version on startup, and run the appropriate migration tools? (or, prompt the user to run them)

jbenet · 2015-04-07T06:16:59Z

@whyrusleeping for now prompt the user to run them. we can do things automatically later, but if something goes wrong for now, would be ideal to have the user on top of it

whyrusleeping · 2015-04-07T21:27:47Z

im not convinced we should have a tool for each migration, i think a single tool would be the better approach

jbenet · 2015-04-08T09:00:51Z

Need to make sure we can rest assured these will work years from now without our maintenance. The repo's code will progress, often change fundamentally. Forcing us to maintain things backwards compatible with all repo versions -- or even one tool -- is signing us up for pain in the future, testing with old things, etc. Instead, if we freeze its code now, with its own vendoring, we can count on these working with the current repo versions down the road.

I'm envisioning a pretty simple setup where:

fs-repo-migrations/
├── 0-to-1/
│   ├── all_the_code_needed_for_0-to-1/
│   └── main.go
├── 1-to-2/
│   ├── all_the_code_needed_for_1-to-2/
│   └── main.go
└── 2-to-3/
    ├── all_the_code_needed_for_2-to-3/
    └── main.go

so we can go into any of these, compile and know it will work.

depending on how we do things, we may not need to use godeps, we could just vendor directly (cp -r) what we need into those directories.

jbenet · 2015-04-09T06:35:33Z

repo/fsrepo/fsrepo.go

+	vers := string(ver)[:1]
+
+	if vers != RepoVersion {
+		return nil, fmt.Errorf("Repo has incorrect version: '%s'\nProgram version is: '%s'\nPlease run the appropriate migration tool before continuing",


var MsgUpgradeRepo = `Repo has incorrect version: %s
Program version is: %s
Please run the repo migration tool before continuing:

<cmd for them to run>

`

I'll think about what a good way to do <cmd for them to run> is. the problem is they:

may not have it installed (installed ipfs as a binary).

may not have go installed (may not be able to compile).

do not have ipfs installed (cannot ipfs cat <hash>/<arch>/1-to-2 >1-to-2)

May be able to do something like:

wget http://gateway.ipfs.io/ipfs/<hash>/<arch>/1-to-2 chmod +x 1-to-2 ./1-to-2

And ship a little shell script that does it as:

ipfs-repo-migrate 1-to-2

go 1.5 is coming at such a good moment...

we could at least just have a page somewhere discussing migrations, maybe a github repo, and link them to it in the error message

jbenet · 2015-04-09T06:58:13Z

@whyrusleeping LGTM!! I'll write the script to download the migrations for us.

jbenet · 2015-04-10T00:27:07Z

cmd/ipfs/daemon.go

-	// _before_ acquiring the daemon lock so the user gets an appropriate error
-	// message.
-	// NB: It's safe to read the config without the daemon lock, but not safe
-	// to write.


@whyrusleeping will this still yield the right error?

the errors returned may now differ, but they actually mean something now. the other error was completely worthless

jbenet · 2015-04-10T00:36:13Z

@whyrusleeping the new changes may alter the semantics of ipfs startup. are the right errors still gotten?

I'll start full tests manually.

jbenet · 2015-04-10T10:23:14Z

Every commit passed for me. And i think for @whyrusleeping as well.

jbenet · 2015-04-10T18:27:55Z

i tried to avoid using go-ipfs code when i could (minus a few repo imports) to make this easier.

+1. We should probably vendor those just in case :| -- we dont have to use godeps. we can just copy-paste. (:open_mouth:!)

whyrusleeping · 2015-04-10T18:49:29Z

im fine with copy pasting, especially since we are 'vendoring' with the explicit goal of never changing that code

jbenet · 2015-04-12T15:09:01Z

@whyrusleeping what's missing here? just the guide to upgrade?

whyrusleeping · 2015-04-12T18:32:04Z

Just the guide.

jbenet · 2015-04-18T13:21:43Z

Migrations CR

error needs to be friendlier

now:

Error: ipfs repo found in old '.go-ipfs' location, please run migration tool

should be something like:

Error: ipfs repo found in old '.go-ipfs' location, please run migration tool:

    wget http://gateway.ipfs.io/ipfs/<hash>/<go-platform>/ipfs-fsrepo-migrations
    chmod +x ipfs-fsrepo-migrations
    ./ipfs-fsrepo-migrations

If you encounter problems, please see https://github.com/ipfs/fs-repo-migrations/blob/master/run.md

This should work on every machine with as few steps as possible. We could detect if Go is installed, and suggest using go get, but this should only be if they do have it installed... we should not have any go get crap if the machine doesn't have go. We could also use a shell script and pipe straight into shell, but they dont have hashpipe installed and i dont want to do that if they don't.

It's really funny, if ipfs worked during this process, or could install things from it, this would be so easy. lol 🐔 🥚

Hmm. maybe it can be ok to say:

Error: ipfs repo found in old '.go-ipfs' location, please run migration tool.
Please see https://github.com/ipfs/fs-repo-migrations/blob/master/run.md

if we want to make it easy for ourselves this round.

running the migration notes important info -- see 1-to-2 should print to user that repo changed location fs-repo-migrations#1
make https://github.com/ipfs/fs-repo-migrations/blob/master/run.md good. i'm on this
migration worked!! \o/
migration fails mysteriously if daemon running. see migrations should ensure daemon not running fs-repo-migrations#2

jbenet · 2015-04-19T21:23:20Z

On the message, maybe we can say:

Error: ipfs repo found in old '.go-ipfs' location, please run migration tool.
Please see https://github.com/ipfs/fs-repo-migrations/blob/master/run.md
Sorry for the inconvenience. In the future, these will run automatically.

jbenet · 2015-04-20T06:31:47Z

mv daemon.lock -> repo.lock

FSRepo.Open is dead since fdd1cd8

This allows replacing the datastore without needing to write Close through to every wrapped datastore.

WARNING: No migration performed! That needs to come in a separate commit, perhaps amended into this one. Migration must move keyspace "/b" from leveldb to the flatfs subdir, while removing the "b" prefix (keys should start with just "/").

This changes .go-ipfs to .ipfs everywhere. And by the way this defines a DefaultPathName const for this name. License: MIT Signed-off-by: Christian Couder <chriscool@tuxfamily.org>

Improved the repo migration errors to provide instructions to the user.

The "daemon.lock" was really a repo.lock, as the cli also took it and the purpose was any process mutex. This is part of the 1-to-2 migration, and has already been handled in https://github.com/ipfs/fs-repo-migrations/tree/master/ipfs-1-to-2

jbenet · 2015-04-20T09:41:36Z

rebased on new master

fsrepo migrations

jbenet added the status/in-progress In progress label Mar 20, 2015

jbenet force-pushed the migrations branch from e2fc883 to 7d40540 Compare March 20, 2015 14:04

whyrusleeping added topic/repo Topic repo topic/cleanup Topic cleanup labels Mar 28, 2015

jbenet modified the milestone: Flat Repo and More Mar 29, 2015

whyrusleeping assigned jbenet Mar 29, 2015

jbenet mentioned this pull request Mar 29, 2015

config: change default config dir name to .ipfs #975

Closed

jbenet added backlog and removed status/in-progress In progress labels Apr 1, 2015

jbenet added codereview and removed backlog labels Apr 1, 2015

whyrusleeping force-pushed the migrations branch from 7d40540 to d77c58c Compare April 7, 2015 06:06

whyrusleeping force-pushed the migrations branch from d77c58c to 3e6ea00 Compare April 7, 2015 06:27

jbenet reviewed Apr 9, 2015
View reviewed changes

jbenet reviewed Apr 10, 2015
View reviewed changes

jbenet added the backwards incompatible: repo label Apr 13, 2015

jbenet and others added 15 commits April 20, 2015 02:20

0-to-1 fsrepo migration

7eb4a92

LevelDB is no longer "the" datastore, adjust identifiers

5cb8d80

Remove comment referring to old code

26cebac

FSRepo.Open is dead since fdd1cd8

Let FSRepo Close know explicitly about LevelDB

59aa209

This allows replacing the datastore without needing to write Close through to every wrapped datastore.

Update vendored go-datastore, now has flatfs and mount

4f4b894

remove / prefix and close directories properly in query

e965c53

add 1-to-2 migration program

2c79e5d

no longer worry about moving key out of config file

f3fbedf

config: change default config dir name to .ipfs

96a22c5

This changes .go-ipfs to .ipfs everywhere. And by the way this defines a DefaultPathName const for this name. License: MIT Signed-off-by: Christian Couder <chriscool@tuxfamily.org>

make ipfs understand the new migration

c419a48

remove migrations code from go-ipfs, in favor of fs-repo-migrations repo

591cca9

put mfsr package back

bb7bf81

repo: clean up migration errors

c1b8d29

Improved the repo migration errors to provide instructions to the user.

repo: move daemon.lock -> repo.lock

dd25a75

The "daemon.lock" was really a repo.lock, as the cli also took it and the purpose was any process mutex. This is part of the 1-to-2 migration, and has already been handled in https://github.com/ipfs/fs-repo-migrations/tree/master/ipfs-1-to-2

jbenet force-pushed the migrations branch from 46fc27b to 887559a Compare April 20, 2015 09:41

jbenet force-pushed the migrations branch from 887559a to dd25a75 Compare April 20, 2015 09:44

jbenet added a commit that referenced this pull request Apr 20, 2015

Merge pull request #950 from ipfs/migrations

7e887b9

fsrepo migrations

jbenet merged commit 7e887b9 into master Apr 20, 2015

jbenet removed the codereview label Apr 20, 2015

jbenet deleted the migrations branch April 20, 2015 10:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fsrepo migrations #950

fsrepo migrations #950

jbenet commented Mar 20, 2015

jbenet commented Mar 20, 2015

whyrusleeping commented Mar 21, 2015

jbenet commented Apr 1, 2015

whyrusleeping commented Apr 1, 2015

jbenet commented Apr 1, 2015

whyrusleeping commented Apr 1, 2015

whyrusleeping commented Apr 2, 2015

whyrusleeping commented Apr 7, 2015

jbenet commented Apr 7, 2015

whyrusleeping commented Apr 7, 2015

jbenet commented Apr 8, 2015

jbenet Apr 9, 2015

jbenet Apr 9, 2015

whyrusleeping Apr 9, 2015

jbenet commented Apr 9, 2015

jbenet Apr 10, 2015

whyrusleeping Apr 10, 2015

jbenet commented Apr 10, 2015

jbenet commented Apr 10, 2015

jbenet commented Apr 10, 2015

whyrusleeping commented Apr 10, 2015

jbenet commented Apr 12, 2015

whyrusleeping commented Apr 12, 2015

jbenet commented Apr 18, 2015

jbenet commented Apr 19, 2015

jbenet commented Apr 20, 2015

jbenet commented Apr 20, 2015

fsrepo migrations #950

fsrepo migrations #950

Conversation

jbenet commented Mar 20, 2015

jbenet commented Mar 20, 2015

whyrusleeping commented Mar 21, 2015

jbenet commented Apr 1, 2015

whyrusleeping commented Apr 1, 2015

jbenet commented Apr 1, 2015

whyrusleeping commented Apr 1, 2015

whyrusleeping commented Apr 2, 2015

whyrusleeping commented Apr 7, 2015

jbenet commented Apr 7, 2015

whyrusleeping commented Apr 7, 2015

jbenet commented Apr 8, 2015

jbenet Apr 9, 2015

Choose a reason for hiding this comment

jbenet Apr 9, 2015

Choose a reason for hiding this comment

whyrusleeping Apr 9, 2015

Choose a reason for hiding this comment

jbenet commented Apr 9, 2015

jbenet Apr 10, 2015

Choose a reason for hiding this comment

whyrusleeping Apr 10, 2015

Choose a reason for hiding this comment

jbenet commented Apr 10, 2015

jbenet commented Apr 10, 2015

jbenet commented Apr 10, 2015

whyrusleeping commented Apr 10, 2015

jbenet commented Apr 12, 2015

whyrusleeping commented Apr 12, 2015

jbenet commented Apr 18, 2015

Migrations CR

jbenet commented Apr 19, 2015

jbenet commented Apr 20, 2015

jbenet commented Apr 20, 2015