Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

TEM-155: network routing scheme #13

Merged
merged 97 commits into from Jan 18, 2019

Conversation

@bobheadxi
Copy link
Contributor

bobheadxi commented Dec 28, 2018

馃懛 Purpose

See ticket. In a nutshell, we want to provide an interface as follows:

requests for networks go via HTTP to:

  • <orchestrator_address>/networks/<my_network>/gateway
  • <orchestrator_address>/networks/<my_network>/api

etc. etc. the orchestrator daemon should then route requests to the appropriate node.

this opens the door for a wide range of things, like improved availability and hosting across multiple servers, as well as access tracking and potentially limiting

馃殌 Changes

  • new delegator package and server
  • orchestrator::Registry is now public
  • fix for potentially unreleased lock on node startup (41761a5)
  • slightly improved logging in ipfs
  • new commands under ipfs-orchestrator -dev dev
  • new endpoint, NetworkDiagnostics
  • lots of new make shortcuts for ctl commands
  • improved bootstrapping
  • updated for new db

Testing

make testenv
make daemon

set up a node called test_network:

make new-network
make start-network

access the API:

export TOKEN="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1NDc3Nzg2MzcsImlkIjoidGVzdHVzZXIiLCJvcmlnX2lhdCI6MTU0NzY5MjIzN30.2oqQCym2mcyFl8mjOHoGNtK41SJLwX0xbWScDruDECQ"
# scheme: <addr>/network/<name>/<feature>/<request>
curl --header "Authorization: Bearer $TOKEN" 127.0.0.1:8080/network/test_network/api/v0/repo/stat

you should get a response:

{"RepoSize":55956,"StorageMax":100000000000,"NumObjects":15,"RepoPath":"/data/ipfs","Version":"fs-repo@7"}

TODO

  • verify swarm works UPDATE: only because still direct access :(
  • test everything
  • cache proxies
  • (?) make node ports private (on 127.0.0.1 instead of 0.0.0.0) UPDATE: only for gateway, api

鈿狅笍 Breaking Changes

hopefully none

@bobheadxi bobheadxi added the pr: wip label Dec 28, 2018
@bobheadxi

This comment has been minimized.

Copy link
Contributor Author

bobheadxi commented Dec 28, 2018

paging @postables for initial thoughts

@bonedaddy

This comment has been minimized.

Copy link
Member

bonedaddy commented Dec 31, 2018

@bobheadxi sorry about the delay, didn't get a notification for some reason.

I think making ports private on 127.0.0.1 is an excellent idea, and should provide some good security measures as well. I think these are great changes to start implementing. I like being able to route to different networks will make management of these a lot easier, and avoid quirky port forwarding, reverse proxying that isn't built in,

I got an error trying out your example curl command after spinning up the network curl 127.0.0.1:8000/network/test_network/api/v0/repo/stat

2018-12-30T22:57:26.385-0800    INFO    daemon  daemon/daemon.go:103    spinning up server      {"version": "v0.3.1-21-gb6212ee", "host": "127.0.0.1", "port": "9111"}
2018-12-30T23:06:24.521-0800    INFO    delegator.requests      log/middleware.go:33    request completed       {"version": "v0.3.1-21-gb6212ee", "path": "/network/test_network/api/v0/repo/stat", "query": "", "method": "GET", "user-agent": "curl/7.58.0", "status": 404, "took": "36.388碌s", "real-ip": "127.0.0.1:58470", "request-id": "dark/BAxIhXJo2k-000001"}

Not sure if its due to the network not being started as the start or the network creation didn't log anything on the orchestrator side

solidity@dark:~/go/src/github.com/RTradeLtd/ipfs-orchestrator$ ./ipfs-orchestrator --config config.example.json --dev dev ctl StartNetwork Network=test_network
[WARNING] dev mode enabled
@bobheadxi

This comment has been minimized.

Copy link
Contributor Author

bobheadxi commented Jan 10, 2019

@postables did you explicitly start up a new network?

@bonedaddy

This comment has been minimized.

Copy link
Member

bonedaddy commented Jan 10, 2019

@bobheadxi yea i did, ill try again tomorrow after pulling in your most recent updates.

@bonedaddy

This comment has been minimized.

Copy link
Member

bonedaddy commented Jan 10, 2019

@bobheadxi running into an issue:

solidity@dark:~/go/src/github.com/RTradeLtd/ipfs-orchestrator$ ./ipfs-orchestrator -dev dev -config config.example.json db test_network
[WARNING] dev mode enabled

doesn't look like that gives any output, tried with ctl CreateNetwork and it gave a panic

solidity@dark:~/go/src/github.com/RTradeLtd/ipfs-orchestrator$ ./ipfs-orchestrator -dev -config config.example.json ctl CreateNetwork test_network
[WARNING] dev mode enabled
function CreateNetwork found
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x80 pc=0x509620]

goroutine 1 [running]:
github.com/RTradeLtd/ipfs-orchestrator/vendor/github.com/RTradeLtd/ctl.(*CTL).Exec(0xc000335b40, 0xdcd940, 0xc0000b2010, 0xc0000bc050, 0x2, 0x2, 0xdc75c0, 0xc0000b6008, 0x0, 0x0, ...)
        /home/solidity/go/src/github.com/RTradeLtd/ipfs-orchestrator/vendor/github.com/RTradeLtd/ctl/ctl.go:56 +0x360
main.runCTL(0x7ffd67bd3106, 0x13, 0xbd0001, 0xc0000bc050, 0x2, 0x2)
        /home/solidity/go/src/github.com/RTradeLtd/ipfs-orchestrator/ctl.go:43 +0x2ac
main.main()
        /home/solidity/go/src/github.com/RTradeLtd/ipfs-orchestrator/main.go:76 +0x260
@bobheadxi bobheadxi added pr: review and removed pr: wip labels Jan 17, 2019
@bobheadxi bobheadxi requested a review from bonedaddy Jan 17, 2019
@RTradeLtd RTradeLtd deleted a comment from codecov bot Jan 17, 2019
Copy link
Member

bonedaddy left a comment

Amazing! 馃敟

@bobheadxi bobheadxi merged commit 0f82b22 into master Jan 18, 2019
5 checks passed
5 checks passed
Travis CI - Branch Build Passed
Details
Travis CI - Pull Request Build Passed
Details
codecov/patch 84.87% of diff hit (target 50%)
Details
codecov/project No report found to compare against
Details
guardrails/scan no new security issues detected
Details
@bobheadxi bobheadxi deleted the delegator branch Jan 18, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can鈥檛 perform that action at this time.