vendor libp2p with gx #2256

Merged
merged 30 commits into from Feb 6, 2016

Conversation

Projects
None yet
7 participants
@whyrusleeping
Member

whyrusleeping commented Jan 27, 2016

The biggest part of this PR will be making sure everything is still easy to dev with

@rht

This comment has been minimized.

Show comment
Hide comment
@rht

rht Jan 29, 2016

Contributor

(this should take <=~O(1day) the longest)

Contributor

rht commented Jan 29, 2016

(this should take <=~O(1day) the longest)

@diasdavid

This comment has been minimized.

Show comment
Hide comment
Member

diasdavid commented Jan 29, 2016

image

@diasdavid

This comment has been minimized.

Show comment
Hide comment
@diasdavid

diasdavid Jan 29, 2016

Member


Seal of Approval :D (//cc @RichardLitt)

Member

diasdavid commented Jan 29, 2016


Seal of Approval :D (//cc @RichardLitt)

@whyrusleeping

This comment has been minimized.

Show comment
Hide comment
@whyrusleeping

whyrusleeping Jan 29, 2016

Member

i still need to fix the tests so they know what the hell is going on

Member

whyrusleeping commented Jan 29, 2016

i still need to fix the tests so they know what the hell is going on

whyrusleeping added some commits Jan 27, 2016

initial vendoring of libp2p outside of the repo with gx
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
package.json cleanup
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
add makefile rule for grabbing gx and deps
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
go-keyspace dep from libp2p added
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
correct go-log dep
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
add better makefile and reference gx stuff in readme
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
go go circle tests!
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
circle.yml fix?
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
verbose gx install
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
gx install stuff in cmd/ipfs/Makefile
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
circleci verbose
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
travis file change
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
circleci fun fun fun
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
i'm beginning to dislike circleci
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
on a scale from 1 to annoying, i don't like circleCI anymore at all
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
ciiiii issss awesome
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
circleci has bad documentation
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
lets not have any documentation for this stupid file format, its awesome
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
also, yaml is a terrible format
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
documentation saves lives
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
update libp2p dep
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
do that last thing again
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@whyrusleeping

This comment has been minimized.

Show comment
Hide comment
@whyrusleeping

whyrusleeping Feb 1, 2016

Member

now i'm running into docker issues... id like to copy $GOPATH/src/gx into the build context, but i dont see any easy way to do that. Its looking like i'll have to install gx on the docker image and run an install... which is annoying because i've already done it once during the CI test runs...

cc @lgierth

Member

whyrusleeping commented Feb 1, 2016

now i'm running into docker issues... id like to copy $GOPATH/src/gx into the build context, but i dont see any easy way to do that. Its looking like i'll have to install gx on the docker image and run an install... which is annoying because i've already done it once during the CI test runs...

cc @lgierth

lgierth added some commits Feb 4, 2016

Rework the Dockerfile
- Have two Dockerfiles doing essentially the same,
  but optimized for build time (for tests)
  and image size (for Docker Hub)
- Fetch gx dependencies
- Expose port 4002 for utp
- Specify go version, currently 1.5.3-r0
- Create ephemeral fs-repo if none is mounted
- Have t0300-docker-image actually test IPFS, not just an echo
- Make everything a bit less hardcoded
- Remove dead shacheck

License: MIT
Signed-off-by: Lars Gierth <larsg@systemli.org>
sharness: disable t0300-docker-image on Circle CI
The actual tests for the IPFS daemon within the image,
which were added in the previous commits, made Circle CI unhappy.

Circle CI runs an old version of Docker
which still uses LXC instead of libcontainer.
The existing lxc-attach hack is testament to that.

We might be able to get it work [1][2]
but it's really not worth it at the moment,
and Circle CI isn't a good option for Docker things, right now.

[1] https://jpetazzo.github.io/2014/03/23/lxc-attach-nsinit-nsenter-docker-0-9/
[2] https://github.com/jpetazzo/nsenter

License: MIT
Signed-off-by: Lars Gierth <larsg@systemli.org>
docker: save image space by excluding .git/
License: MIT
Signed-off-by: Lars Gierth <larsg@systemli.org>
@lgierth

This comment has been minimized.

Show comment
Hide comment
@lgierth

lgierth Feb 5, 2016

Member

Hey @Luzifer please take a look at my docker-related commits here. It's a bit more involved now, we can tune it in another PR -- this here is just to get gx integration going.

Member

lgierth commented Feb 5, 2016

Hey @Luzifer please take a look at my docker-related commits here. It's a bit more involved now, we can tune it in another PR -- this here is just to get gx integration going.

@lgierth

This comment has been minimized.

Show comment
Hide comment
@lgierth

lgierth Feb 5, 2016

Member

@whyrusleeping it's good now

Member

lgierth commented Feb 5, 2016

@whyrusleeping it's good now

@whyrusleeping

This comment has been minimized.

Show comment
Hide comment
@whyrusleeping

whyrusleeping Feb 5, 2016

Member

oh my god look those tests are green o.o

Member

whyrusleeping commented Feb 5, 2016

oh my god look those tests are green o.o

@whyrusleeping

This comment has been minimized.

Show comment
Hide comment
@whyrusleeping

whyrusleeping Feb 5, 2016

Member

@RichardLitt can you take a look at the usability here? Maybe help me out making the readme clear on how this works (all anyone should have to do is run make install)

Member

whyrusleeping commented Feb 5, 2016

@RichardLitt can you take a look at the usability here? Maybe help me out making the readme clear on how this works (all anyone should have to do is run make install)

@whyrusleeping

This comment has been minimized.

Show comment
Hide comment
@whyrusleeping

whyrusleeping Feb 5, 2016

Member

i'll merge this in the morning, we can always improve the readme in a followup PR

Member

whyrusleeping commented Feb 5, 2016

i'll merge this in the morning, we can always improve the readme in a followup PR

@chriscool

This comment has been minimized.

Show comment
Hide comment
@chriscool

chriscool Feb 5, 2016

Contributor

In the sharness tests some work has been done to remove any dependency on wget and to use curl instead. It would be nice here to use curl as much as possible instead of wget.

Contributor

chriscool commented Feb 5, 2016

In the sharness tests some work has been done to remove any dependency on wget and to use curl instead. It would be nice here to use curl as much as possible instead of wget.

@lgierth

This comment has been minimized.

Show comment
Hide comment
@lgierth

lgierth Feb 5, 2016

Member

@chriscool wget is being used inside the container. I tried curl first but it doesn't have the retrying options I was looking for (retrying unconditionally).

Member

lgierth commented Feb 5, 2016

@chriscool wget is being used inside the container. I tried curl first but it doesn't have the retrying options I was looking for (retrying unconditionally).

@rht

This comment has been minimized.

Show comment
Hide comment
@rht

rht Feb 5, 2016

Contributor

(in other situation, ipfs get/ipget was being used in place of curl since the latter doesn't do recursive download (while wget does) )

Contributor

rht commented Feb 5, 2016

(in other situation, ipfs get/ipget was being used in place of curl since the latter doesn't do recursive download (while wget does) )

@RichardLitt

This comment has been minimized.

Show comment
Hide comment
@RichardLitt

RichardLitt Feb 5, 2016

Member

@whyrusleeping I opened github.com/ipfs/go-ipfs/pull/2302 to merge into this branch. Is that what you wanted?

Member

RichardLitt commented Feb 5, 2016

@whyrusleeping I opened github.com/ipfs/go-ipfs/pull/2302 to merge into this branch. Is that what you wanted?

RichardLitt and others added some commits Feb 5, 2016

Cleaned up README
I did the following:
- Capitalized IPFS wherever it did not explicitely refer to the CLI command
- Moved Dev Dependencies into the Install section
- Added a Table of Contents
- Capitalised and added periods to various sentences
- Capitalised Go
- Removed double empty lines
- Reindented a few sections to make the flow more clear.
- Added a note to the Install section about how to canonically install.

License: MIT
Signed-off-by: Richard Littauer <richard.littauer@gmail.com>
assets/assets_test.go
@@ -9,10 +9,12 @@ import (
// TestEmbeddedDocs makes sure we don't forget to regenerate after documentation change
func TestEmbeddedDocs(t *testing.T) {
+ t.Skip("skipping for now, paths are hard")

This comment has been minimized.

@noffle

noffle Feb 5, 2016

Contributor

I'm not really familiar with assets/: what is this test supposed to do and why are we now skipping it? Could we get a tracking issue and link to it here so it doesn't fall entirely off the radar?

@noffle

noffle Feb 5, 2016

Contributor

I'm not really familiar with assets/: what is this test supposed to do and why are we now skipping it? Could we get a tracking issue and link to it here so it doesn't fall entirely off the radar?

This comment has been minimized.

@whyrusleeping

whyrusleeping Feb 5, 2016

Member

the test is testing that the string constants in this file match the files stored in git. Except now those files are stored in gx (and we arent storing the gx deps in git). I can probably add a bit of logic to search for the installed package either locally or globally. Good catch

@whyrusleeping

whyrusleeping Feb 5, 2016

Member

the test is testing that the string constants in this file match the files stored in git. Except now those files are stored in gx (and we arent storing the gx deps in git). I can probably add a bit of logic to search for the installed package either locally or globally. Good catch

@noffle

This comment has been minimized.

Show comment
Hide comment
@noffle

noffle Feb 5, 2016

Contributor

Nice work @whyrusleeping. It's been cool watching gx go from proof of concept to powering go-ipfs.

Some usability feedback:

  1. make in cmd/ipfs/ doesn't work: gx looks for package.json, which of course is back in the root directory.
  2. make is now very expensive. Every invocation does go get -u gx, go get -u gx-go, and gx install, even when you're up to date. This makes my builds internet dependent, and significantly slower -- adds ~43 seconds. Other folks might have different workflows, but I rely on make for development; I'd be really keen on keeping it fast. Maybe having dummy files we touch on go get and gx install would work well with make?
  3. More makefile stuff: right now only install depends on deps, but I think build targets should rely on deps instead -- and then we can have install rely on build. Does that make sense?

Kudos @RichardLitt for the nice clean-up to the README; it's an entry point for so many people who look at IPFS. 👍

Contributor

noffle commented Feb 5, 2016

Nice work @whyrusleeping. It's been cool watching gx go from proof of concept to powering go-ipfs.

Some usability feedback:

  1. make in cmd/ipfs/ doesn't work: gx looks for package.json, which of course is back in the root directory.
  2. make is now very expensive. Every invocation does go get -u gx, go get -u gx-go, and gx install, even when you're up to date. This makes my builds internet dependent, and significantly slower -- adds ~43 seconds. Other folks might have different workflows, but I rely on make for development; I'd be really keen on keeping it fast. Maybe having dummy files we touch on go get and gx install would work well with make?
  3. More makefile stuff: right now only install depends on deps, but I think build targets should rely on deps instead -- and then we can have install rely on build. Does that make sense?

Kudos @RichardLitt for the nice clean-up to the README; it's an entry point for so many people who look at IPFS. 👍

@whyrusleeping

This comment has been minimized.

Show comment
Hide comment
@whyrusleeping

whyrusleeping Feb 5, 2016

Member

@noffle good catch on the cmd/ipfs makefile, missed that one. As for make being expensive, I do want to ensure that anyone building ipfs as per the listed build instructions has their deps up to date. Building without the makefile is quite easy: go install ./cmd/ipfs or go build ./cmd/ipfs. We could have a make fast-install or make fast-build type target for your usecase if you don't want to type the go commands.

having install depend on build depend on deps is a good idea. changing that now.

Member

whyrusleeping commented Feb 5, 2016

@noffle good catch on the cmd/ipfs makefile, missed that one. As for make being expensive, I do want to ensure that anyone building ipfs as per the listed build instructions has their deps up to date. Building without the makefile is quite easy: go install ./cmd/ipfs or go build ./cmd/ipfs. We could have a make fast-install or make fast-build type target for your usecase if you don't want to type the go commands.

having install depend on build depend on deps is a good idea. changing that now.

makefile fixes
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@noffle

This comment has been minimized.

Show comment
Hide comment
@noffle

noffle Feb 5, 2016

Contributor

yay 👍 for reduced redundancy

Contributor

noffle commented on cmd/ipfs/Makefile in c9b83c5 Feb 5, 2016

yay 👍 for reduced redundancy

@noffle

This comment has been minimized.

Show comment
Hide comment
@noffle

noffle Feb 5, 2016

Contributor

nofuse needs deps too

Contributor

noffle commented on Makefile in c9b83c5 Feb 5, 2016

nofuse needs deps too

whyrusleeping added some commits Feb 6, 2016

note in assets test
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
add deps target to nofuse
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@noffle

This comment has been minimized.

Show comment
Hide comment
@noffle

noffle Feb 6, 2016

Contributor

As for make being expensive, I do want to ensure that anyone building ipfs as per the listed build instructions has their deps up to date.

Agreed. Ideally we'd have versions of gx and gx-go pinned so we could skip these heavy steps. I'll think about the problem some more, but yeah: it doesn't make sense to hold up this PR because of it.

Contributor

noffle commented Feb 6, 2016

As for make being expensive, I do want to ensure that anyone building ipfs as per the listed build instructions has their deps up to date.

Agreed. Ideally we'd have versions of gx and gx-go pinned so we could skip these heavy steps. I'll think about the problem some more, but yeah: it doesn't make sense to hold up this PR because of it.

@noffle

This comment has been minimized.

Show comment
Hide comment
@noffle

noffle Feb 6, 2016

Contributor

🐴 LGTM

Contributor

noffle commented Feb 6, 2016

🐴 LGTM

whyrusleeping added a commit that referenced this pull request Feb 6, 2016

@whyrusleeping whyrusleeping merged commit 5a0b802 into master Feb 6, 2016

1 of 5 checks passed

continuous-integration/travis-ci/pr The Travis CI build could not complete due to an error
Details
continuous-integration/travis-ci/push The Travis CI build could not complete due to an error
Details
continuous-integration/appveyor/branch AppVeyor build failed
Details
continuous-integration/appveyor/pr AppVeyor build failed
Details
ci/circleci Your tests passed on CircleCI!
Details

@whyrusleeping whyrusleeping deleted the feat/gx-libp2p branch Feb 6, 2016

@rht

This comment has been minimized.

Show comment
Hide comment
@rht

rht Feb 6, 2016

Contributor

Somehow missed the go get -u in the make install. This breaks the immutability of the build of go-ipfs of a specific commit (with gx remains as being mutable).

Contributor

rht commented Feb 6, 2016

Somehow missed the go get -u in the make install. This breaks the immutability of the build of go-ipfs of a specific commit (with gx remains as being mutable).

@noffle

This comment has been minimized.

Show comment
Hide comment
@noffle

noffle Feb 6, 2016

Contributor

@rht: yeah +1. I'd like to see a means of retrieving a pinned version of gx and gx-go as well. Maybe via ipget or just wgeting the public gateway?

Contributor

noffle commented Feb 6, 2016

@rht: yeah +1. I'd like to see a means of retrieving a pinned version of gx and gx-go as well. Maybe via ipget or just wgeting the public gateway?

@RichardLitt RichardLitt referenced this pull request in ipfs/ipfs Feb 6, 2016

Open

IPFS Weekly Updates #151

@rht

This comment has been minimized.

Show comment
Hide comment
@rht

rht Feb 7, 2016

Contributor

It is simplest to just bootstrap gx, gx-go in vendor/ (this is the closest thing to hash/version pinning in go anyway, albeit implicit). That way, the deps that are fetched by gx are still pinned by their hashes or versions from package.json.

Contributor

rht commented Feb 7, 2016

It is simplest to just bootstrap gx, gx-go in vendor/ (this is the closest thing to hash/version pinning in go anyway, albeit implicit). That way, the deps that are fetched by gx are still pinned by their hashes or versions from package.json.

@noffle noffle referenced this pull request in ipfs/pm Feb 8, 2016

Closed

Sprint: February 1 #88

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment