Skip to content
Please note that GitHub no longer supports Internet Explorer.

We recommend upgrading to the latest Microsoft Edge, Google Chrome, or Firefox.

Learn more
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add docker support #745

Merged
merged 2 commits into from Jan 24, 2020
Merged

Conversation

@ldoublewood
Copy link
Contributor

ldoublewood commented Dec 5, 2019

  1. Add Dockerfile file, mostly borrow from ipfs project
  2. Add .dockerignore file
  3. Modify Makefile to build extern submodule 'filecoin-ffi' from source instead of download from github release

This PR can close issue: #594

Makefile Outdated Show resolved Hide resolved
@whyrusleeping whyrusleeping requested a review from Kubuxu Dec 5, 2019
@whyrusleeping

This comment has been minimized.

Copy link
Member

whyrusleeping commented Dec 5, 2019

Gonna ask @Kubuxu to do the review here, i'm not good at dockerfiles

@eshon

This comment has been minimized.

Copy link
Contributor

eshon commented Dec 6, 2019

In case it helps @travisperson shared this Docker scripts gist. They've been edited though and not tested after edit so just pasting here for comparison. This one doesn't expose port 5678.

This one is also working for me: https://github.com/mateodelnorte/filecoin-lotus-docker

@whyrusleeping whyrusleeping requested a review from travisperson Dec 8, 2019
@whyrusleeping

This comment has been minimized.

Copy link
Member

whyrusleeping commented Dec 8, 2019

Thanks @eshon! Thats good context to have. I'm tagging @travisperson for review now

@Kubuxu
Kubuxu approved these changes Dec 8, 2019
Copy link
Contributor

Kubuxu left a comment

SGWM but I'm not docker expert.

Copy link
Member

travisperson left a comment

I think this looks fine

@eshon

This comment has been minimized.

Copy link
Contributor

eshon commented Dec 9, 2019

Note that if people have an earlier install they might have docker cache problems so some instructions running clean versions would be great as comments @ldoublewood

@ldoublewood

This comment has been minimized.

Copy link
Contributor Author

ldoublewood commented Dec 9, 2019

@eshon Would you please elaborate what kind of "docker cache problem" will happen?

@eshon

This comment has been minimized.

Copy link
Contributor

eshon commented Dec 9, 2019

@ldoublewood I think if you don't clear your cache it may not pull latest from master but from a saved copy in your cache so may want to run with --no-cache option or whatever else there is, someone on Slack had that problem.

@eshon eshon mentioned this pull request Dec 12, 2019
@whyrusleeping

This comment has been minimized.

Copy link
Member

whyrusleeping commented Dec 19, 2019

@eshon should we add anything to the docs about this? I think we can merge this now in any case

@eshon

This comment has been minimized.

Copy link
Contributor

eshon commented Dec 20, 2019

For now, you guys could create a tools/dockers/docker-examples directory in the lotus repo for community-contributed dockers and docker compose scripts.

@travisperson could use tools/dockers/dockers-infra but let's see what he thinks.

Each community contribution should have a separate descriptive folder of its purpose. For example:

  • tools/dockers/docker-examples/basic-miner-busybox/ (for ldoublewood's here)
  • tools/dockers/docker-examples/dev-api-arch (for mateo's docker compose)
  • tools/dockers/docker-examples/dev-api-debian (for RTrade's docker compose)
  • tools/dockers/docker-examples/local-network-busybox (for a potential local devnet Docker)

tools/dockers/README.md says:

"docker-examples are community-contributed. Have a new type of Docker to share? Make a PR with your scripts or a README.md linking to its home repo and purpose. See a bug? File an Issue so we can keep these up-to-date."


TLDR;

There are different Docker and Docker Compose scripts for different use cases.

Use Cases:

  1. Spinning up a miner
    a. basic storage miner (ldoublewood's here is a good starter for this?)
    b. distributed storage miner (docker compose >> api for seal workers, etc.)
  2. Spinning up API endpoints for building apps
    a. local node for a developer
    b. hosted semi-public endpoint for multiple developers
  3. Spinning up a local network or new network
    a. basic local network (e.g. https://lotu.sh/en+setup-local-dev-net)
    b. new devnet (faucet, genesis, bootstrap, etc.)

Nice-to-have:

  • optional caching of proof params

The dockers in go-filecoin's .docker directory and ones that @travisperson creates are traditionally for use case 3b. They are regularly used and tested by Infra.

However it'd be nice to collect Dockers for all use cases somewhere for the community that's easy to find.

Option 1) Community dockers in a new Docker directory in the lotus repo - Easy for community members to find. Great examples of the nuances of running nodes.

Option 2) filecoin-shipyard/lotus-dockers or filecoin-project/dockers - Less visibility, far from node implementation but in tune with go-filecoin's tradition that repo Dockers are for network infra so just references and actively maintained.

TODO:
Ask Jim to add Docker links to the docs wherever Dockers end up

@jsign

This comment has been minimized.

Copy link
Contributor

jsign commented Jan 4, 2020

A quick run of these Dockerfile seemed to have non-growing volumes. Taking a look inside the container .lotus was living on /root.
Coming back to the Dockerfile: USER lotus seems to be missing just before CMD

@jsign

This comment has been minimized.

Copy link
Contributor

jsign commented Jan 4, 2020

Running with USER lotus has some small problem with /var/tmp/... permissions since lotus daemon fails while downloading and loading the params. The folder is owned by root and users group doesn't have enough permissions! so an extra tweak seems to be needed there.

Maybe some anyone can further check if these facts are correct :)

@whyrusleeping

This comment has been minimized.

Copy link
Member

whyrusleeping commented Jan 24, 2020

Gonna merge this now, lots of people have been asking for it. Its probably not perfect yet, but we do accept PRs :)

@whyrusleeping whyrusleeping merged commit 9da4296 into filecoin-project:master Jan 24, 2020
6 checks passed
6 checks passed
ci/circleci: build-all Your tests passed on CircleCI!
Details
ci/circleci: lint-changes Your tests passed on CircleCI!
Details
ci/circleci: mod-tidy-check Your tests passed on CircleCI!
Details
ci/circleci: test Your tests passed on CircleCI!
Details
codecov/patch Coverage not affected when comparing a45fe8a...86f078d
Details
codecov/project 30.16% remains the same compared to a45fe8a
Details
@jsign jsign mentioned this pull request Jan 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.