Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate pinning / seeding daemon with CLI #1081

17 of 27 tasks
RangerMauve opened this issue Mar 26, 2019 · 6 comments
17 of 27 tasks

Integrate pinning / seeding daemon with CLI #1081

RangerMauve opened this issue Mar 26, 2019 · 6 comments


Copy link

@RangerMauve RangerMauve commented Mar 26, 2019

I'm using this to track progress on the work I'll be doing integrating a pinning / seeding service with the Dat CLI.

  • Figure out what to call it so it makes sense to everyone (twitter poll)
    • Going to go with pinning for now since that's the name of the DEP
  • Identify what to use for the pinning
  • Should integrate with DEP 0003 for an HTTP API (local only, should it have auth?)
  • Build a cli extension: dat-pin
    • Persist credentials / service URL to FS
    • Commands to talk to service
      • set-service / unset-service / get-service
      • login / logout
      • add / remove / list
    • Local pinning service
      • start-service, install-service, uninstall-service
      • Use os-service module
        • Linux
        • Mac
        • Windows
      • Integrate with discovery-swarm-web so that the daemon also acts as a proxy for web apps.
  • Integrate with dat CLI pr
  • Make sure it works with executible dat cli
  • Update docs with examples on how to use it
  • Figure out a DEP for different data structures to identify public keys of additional feeds that need to be replicated
  • Add support for this DEP to the pinning service
  • Add docs for spinning up Homebase using Docker or Digital Ocean / AWS
Copy link

@joehand joehand commented Mar 26, 2019

If you want to start with any of these as an extension we can add them, they just need to be published as dat-something and then whitelisted. Then can be run as top level commands dat something.

Copy link
Contributor Author

@RangerMauve RangerMauve commented Mar 26, 2019

I was thinking I'd need to modify the existing registry code so that dat publish would automatically target a daemon. Though launching the daemon could easily be a separate module.

Copy link
Contributor Author

@RangerMauve RangerMauve commented Mar 29, 2019

Here's the progress on the extension: dat-pin.

It's "working" locally, but I'll need to add unit tests and stuff next week before I release it.

Copy link
Contributor Author

@RangerMauve RangerMauve commented Apr 1, 2019

dat-pin is tested and released. Working on adding it to the CLI

@RangerMauve RangerMauve mentioned this issue Apr 1, 2019
Copy link
Contributor Author

@RangerMauve RangerMauve commented Apr 2, 2019

Bikeshedding discussion on IRC:

10:11 <@pfrazee> Paul haha
10:11 going to invite a couple other people. Bikesheds are brutal as open discussions
10:12 <rangermauve> RangerMauve I'll have some tissues on hand in case I break out into tears
10:12 @pfrazee set the topic: Bikesheds suck, let's do this real quick as a closed discussion
10:12 <@pfrazee> Paul ha
10:15 rangermauve: good work on the cli btw
10:15 <rangermauve> RangerMauve Thank you! It was handy that you already made the pinning client and garbados made dat-librarian. I just needed to add a bit of glue and persistence
10:22 → jhand joined (sid114474@gateway/web/
10:22 <@pfrazee> Paul
closepfrazee — 2 Apr 2019
10:23 @pfrazee set the topic: Bikesheds suck, let's do this real quick as a closed discussion. Current list:
10:23 <@pfrazee> Paul so my current ideas are "mirroring service" and "persistence service"
10:24 <rangermauve> RangerMauve I still really like "publishing to a resharing service" TBH
10:24 Persistence and mirroring are pretty good. They sound pretty techy though
10:24 What's your main concern with pinning?
10:25 <@pfrazee> Paul I don't think it's very descriptive
10:25 <rangermauve> RangerMauve I agree. It doesn't relate to what it's actually doing and why it's needed
10:26 <@pfrazee> Paul I'd like to be able to say to somebody "You need a X" and they be like "oh yeah makes sense"
10:26 You need a Permahost
10:26 <rangermauve> RangerMauve What kind of somebody are you targeting? I think that unless they already understand P2P, it'll be hard to find a word for it.
10:27 <@pfrazee> Paul Technical to semi-technical, with the potential for it to become vernacular
10:27 people understand what having a "cloud" means now
10:28 <rangermauve> RangerMauve And at the same time, if they already know about P2P, then they might already be comfortable with words from other projects like IPFS or BitTorrent
10:28 <@pfrazee> Paul like, if you said "oh no somebody hacked my cloud" people would understand that you had some service that hosted your data and it got broken into
10:28 rangermauve: right, that's why I was comfortable with pinning before now
10:28 and it may be our best option still
10:29 <rangermauve> RangerMauve Seeding is also pretty popular due to BT
10:29 <@pfrazee> Paul "Seeding service" is pretty darn clear if you know BT
10:29 "You need a seed"
10:29 <rangermauve> RangerMauve From the poll I ran on twitter it seems those two were pretty close.
10:29 <@pfrazee> Paul oh nice thanks
10:30 <rangermauve> RangerMauve I don't think that either of them are intuitive unless you already know what those words mean in the context of P2P. :P
10:30 <@pfrazee> Paul Emilia's point about the connotations of BitTorrent is pretty accurate
10:30 rangermauve: agree
10:31 "Durability" is an interesting theme to play with
10:33 <rangermauve> RangerMauve Durability seems good, but I think it would also be aimed more towards sysadmins and distributed systems people. I think that outside those contexts it implies that it keeps your Dat from snapping in half somehow
10:33 <@pfrazee> Paul good point...
10:33 okay maybe there's an "Uptime" or "Availability" theme?
10:33 <rangermauve> RangerMauve Same with mirroring, since that's more for people used to file sharing websites that have that
10:33 I think those have the same audience. :P
10:34 Which is OK, IMO
10:34 <@pfrazee> Paul well with uptime you can get an intuitive sensibility for some people, like, it's your remote presence
10:34 the positive about "mirror" is that it suggests it's a dumb thing that's just being a stand-in for you
10:35 <rangermauve> RangerMauve Yeah, I think mirror is also a pretty common word used for something like this.
10:35 <@pfrazee> Paul it's in the "not bad" category for me
10:35 <rangermauve> RangerMauve People thinking of Dat in the context of file sharing would already know what it means to some extent.
10:36 <@pfrazee> Paul it's also got history with websites
10:36 <rangermauve> RangerMauve Yeah!
10:36 Brb, feeding cats
10:36 <@pfrazee> Paul kk
10:37 another idea: "Keeper"
10:39 another idea: "Steward"
10:39 <@pfrazee> Paul another idea: "Base"
10:45 → mafintosh joined (sid28509@gateway/web/
10:49 <rangermauve> RangerMauve I just have this nightmare where Amazon gets into P2P hosting and starts extending things with closed source functionality
10:49 Or some other big company
10:49 <@pfrazee> Paul well IMO, if dat becomes a thing, you should count on Amazon and Google and Microsoft doing that
10:50 pinning services are a hosting resource and they have hosting resources
10:51 <rangermauve> RangerMauve I guess they'll _have_ to add proprietary extensions if they want to differentiate themselves
10:51 <@pfrazee> Paul perhaps. I get that anxiety. It's literally what google is doing right now with AMP on the Web and Email
10:51 <mafintosh> Mathias Buus Hello
10:51 <@pfrazee> Paul mafintosh: hey
10:52 <rangermauve> RangerMauve 👋 Thinking of words that are more intuitive than pinning to people that don't already know what pinning is
10:52 I'm a fan of mirroring and seeding, but those still require you to have a certain background
10:53 <@pfrazee> Paul rangermauve: if we were to focus on there being lots of pinning (or X) services, what I'd feel the need to solve within beaker is a toolset for people to choose providers
10:54 so what I'd want to do is setup arrangements with providers to maintain an SLA that promises they'll stick around in exchange for being recommended by beaker
10:54 <rangermauve> RangerMauve Yeah. Part of the reason I really wanted to get this pinning stuff in the CLI (and the discovery-swarm-web stuff before) was to make the standards easy to use. There's a small bunch of companies working with Dat content, and I think they're all inventing their own pinning services and gateways
10:55 If we had a way to tell them "Here's what a pinning service should look like" then we'd have less fragmentation and something like that would be more feasible
10:55 <@pfrazee> Paul rangermauve: agree, and Im really glad you're doing that. Also, to be clear, Beaker will always make custom providers easy to setup too.
10:56 yes
11:01 <@pfrazee> Paul so, we can circle back to the "many providers vs a few" topic in the future, but from my perspective with beaker, what I need is reliable partners
11:02 <rangermauve> RangerMauve Agreed
11:02 <@pfrazee> Paul I'm going to have beaker include one-click default providers to signup for. I'd be 1000% happy if beaker didnt run them, and if there was more than one
11:04 but pinning providers should *not* have a way to differentiate from each other, and theyre incredibly low margin
11:04 so- let's setup another conversation about that and get darius involved
11:04 <rangermauve> RangerMauve Sounds good. :P
11:05 So all that to say, I think mirroring is a good term because there's usually lots of mirrors for a given file and it's a known term for people that do file sharing but maybe not p2p
11:06 <@pfrazee> Paul yeah, mirroring isnt bad. Let's try a few more terms
11:06 mafintosh: just to check, do you have any bikeshed suggestions on that?
11:07 <mafintosh> Mathias Buus are there any other things other than ipfs that use the word pinning today?
11:07 <@pfrazee> Paul no
11:07 <rangermauve> RangerMauve Should we use an existing term or invent a new one?
11:08 <@pfrazee> Paul I think both should be on the table
11:08 <mafintosh> Mathias Buus mirror is good
11:08 <@pfrazee> Paul (I'm just going to throw out random ideas that come into my head as we talk)
11:08 idea: Courier/carrier
11:09 <rangermauve> RangerMauve I'm personally more into using existing terms since it'll result in fewer people to educate. I'm usually pretty skeptical of P2P/Blockchain projects that invent a bunch of new terms for existing concepts
11:09 <@pfrazee> Paul rangermauve: oh yeah I hear that. 
11:10 <rangermauve> RangerMauve There was one I read recently, and it had like 20 terms and most of them were just existing concepts that got turned into marketing material
11:10 <mafintosh> Mathias Buus pfrazee: courier is no good for non native speakers
11:10 carrier also, they'd think aircraft carriers
11:10 <@pfrazee> Paul mafintosh: yeah. (No need to respond to any of my ideas if they dont land, Im just throwing em out)
11:11 <mafintosh> Mathias Buus :)
11:11 uptime service
11:11 <@pfrazee> Paul I was toying with "uptime" earlier
11:14 <rangermauve> RangerMauve Yeah, so it'd be cool to ~steal~, I mean _borrow_ existing terms
11:14 (I mean, that's why I think it would be cool)
11:15 <@pfrazee> Paul yeah
11:16 <mafintosh> Mathias Buus what are some existing terms? other than mirror
11:16 and hosting
11:16 <@pfrazee> Paul server
11:16 cloud
11:16 <rangermauve> RangerMauve seed, and pin? :P
11:16 <mafintosh> Mathias Buus replica
11:17 <@pfrazee> Paul replica...
11:18 <rangermauve> RangerMauve I think replica is a bit too techy. I don't think everyone knows about databases
11:18 <@pfrazee> Paul node
11:18 <mafintosh> Mathias Buus copy
11:18 <rangermauve> RangerMauve file share?
11:18 <@pfrazee> Paul store
11:18 <mafintosh> Mathias Buus i like store
11:19 <rangermauve> RangerMauve repository (that's what's in the CLI now)
11:19 <mafintosh> Mathias Buus 'can you store this dat for me'
11:19 <@pfrazee> Paul 'store service'
11:19 <rangermauve> RangerMauve storage service?
11:19 <mafintosh> Mathias Buus store sounds friendly to me
11:20 it's the kinda thing you'd say to your neighbor
11:20 <@pfrazee> Paul `dat store add`
11:20 <rangermauve> RangerMauve `dat store set` `dat store dat://foobar`
11:20 <@pfrazee> Paul that might have legs
11:20 <rangermauve> RangerMauve I like it :D
11:21 <@pfrazee> Paul okay let's sit on that for a bit
11:21 rangermauve: if you can just hold off merging your pinning tools into the CLI till we're sure
11:21 <rangermauve> RangerMauve Would you say "Store your dat in a store" or "Store your dat in a storage service" or "storage provider"?
11:21 Yeah, I've still got some work to do around configuration
11:22 Which is what I'm doing now
11:22 <@pfrazee> Paul I think store and storage can be somewhat interchangeable
11:23 but I'd call the service a "dat store" maybe?
11:23 I think this is pretty good
11:23 <rangermauve> RangerMauve What about when there's a marketplace of Dat apps? :P
11:23 I guess it'd be external to the protocol itself
11:24 <@pfrazee> Paul rangermauve: my thinking with beaker is that services and apps will be two different marketplaces with their own sets of sub-types
11:24 "dat stores" being one kind of service
11:24 and beaker will compose them for users
11:24 <rangermauve> RangerMauve Sounds good to me
11:29 <rangermauve> RangerMauve I'll get started on renaming the commands and stuff
11:29 <jhand> Joe Hand cloud friend
11:31 <jhand> Joe Hand can we make the command blue though
11:31 <rangermauve> RangerMauve :P
11:31 ??
11:31 <@pfrazee> Paul jhand: I prefer chartreuse
11:31 rangermauve: bikeshedding jokes
11:32 <rangermauve> RangerMauve Ha ha, that totally went over my head. 😂
11:32 <@pfrazee> Paul okay thanks yall for indulging me on this
11:32 <jhand> Joe Hand I like store!
11:32 <@pfrazee> Paul yeah me too :D
11:32 <rangermauve> RangerMauve Yeah, it's pretty nice
11:32 <@pfrazee> Paul I'll keep everyone updated on the meeting with darius, jhand I'll invite you as well
11:32 <jhand> Joe Hand though that gives us less flexibility around "registry" type pinning services

Mildly redacted to stay on topic


We'll rename dat-pin to dat-store and use dat store dat://somekey along with saying "Put your read key in your dat store".

@pfrazee Maybe you could update the wording on hashbase to align with this term?

Copy link

@pfrazee pfrazee commented Apr 2, 2019

@RangerMauve Yeah I'll have to get that on my todos

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants