Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: implement remote pinning API #3588

Closed
wants to merge 21 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
@@ -328,6 +328,7 @@ module.exports.profiles = profiles
* @property {PubsubConfig} [Pubsub]
* @property {SwarmConfig} [Swarm]
* @property {RoutingConfig} [Routing]
* @property {PinningConfig} [Pinning]
*
* @typedef {Object} AddressConfig
* Contains information about various listener addresses to be used by this node.
@@ -524,4 +525,12 @@ module.exports.profiles = profiles
*
* @typedef {import('ipfs-core-types/src/basic').ToJSON} ToJSON
* @typedef {import('.').AbortOptions} AbortOptions
*
* @typedef {Object} PinningConfig
* @property {Object<string, RemotePinningServiceConfig>} [RemoteServices]
*
* @typedef {Object} RemotePinningServiceConfig
* @property {Object} API
* @property {string} API.Endpoint
* @property {string} API.Key
Copy link
Contributor

@Gozala Gozala Apr 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

go-ipfs settled on omitting endpoint keys when printing config so that they aren't easy to leak (See https://github.com/ipfs/go-ipfs/pull/7661/files#r537641281)

We should at least do the same, although @lidel suggested something smarter is needed in the long term, not sure what exactly he meant but it might be worth doing that here.

Copy link
Member

@lidel lidel Apr 30, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The smarter thing to do is to move all secrets to separate files outside the JSON config.
The only secret right now is Identity.PrivKey this PR adds second type of secret under Pinning.RemoteServices[*].API.Key.
I may be less work to move secrets outside, than to sanitize the output of ipfs config and make sure ipfs config replace does not forget the secrets.

*/
@@ -29,6 +29,7 @@ const createIDAPI = require('./id')
const createConfigAPI = require('./config')
const DagAPI = require('./dag')
const PinManagerAPI = require('./pin/pin-manager')
const PinRemoteAPI = require('./pin/remote')
const createPreloadAPI = require('../preload')
const createMfsPreloadAPI = require('../mfs-preload')
const createFilesAPI = require('./files')
@@ -57,6 +58,8 @@ class IPFS {
constructor ({ print, storage, options }) {
const { peerId, repo, keychain } = storage
const network = Service.create(Network)
const swarm = new SwarmAPI({ network })
const config = createConfigAPI({ repo })

const preload = createPreloadAPI(options.preload)

@@ -86,7 +89,8 @@ class IPFS {
})
const resolve = createResolveAPI({ ipld, name })
const pinManager = new PinManagerAPI({ repo, dagReader })
const pin = new PinAPI({ gcLock, pinManager, dagReader })
const pinRemote = new PinRemoteAPI({ swarm, config, peerId })
const pin = new PinAPI({ gcLock, pinManager, dagReader, pinRemote })
const block = new BlockAPI({ blockService, preload, gcLock, pinManager, pin })
const dag = new DagAPI({ ipld, preload, gcLock, pin, dagReader })
const refs = Object.assign(createRefsAPI({ ipld, resolve, preload }), {
@@ -155,7 +159,7 @@ class IPFS {
this.version = createVersionAPI({ repo })
this.bitswap = new BitswapAPI({ network })
this.bootstrap = new BootstrapAPI({ repo })
this.config = createConfigAPI({ repo })
this.config = config
this.ping = createPingAPI({ network })

this.add = add
@@ -170,7 +174,7 @@ class IPFS {
this.object = new ObjectAPI({ ipld, preload, gcLock, dag })
this.repo = new RepoAPI({ gcLock, pin, repo, refs })
this.stats = new StatsAPI({ repo, network })
this.swarm = new SwarmAPI({ network })
this.swarm = swarm

// For the backwards compatibility
Object.defineProperty(this, 'libp2p', {
@@ -12,15 +12,17 @@ class PinAPI {
* @param {GCLock} config.gcLock
* @param {DagReader} config.dagReader
* @param {PinManager} config.pinManager
* @param {PinRemoteAPI} config.pinRemote
*/
constructor ({ gcLock, dagReader, pinManager }) {
constructor ({ gcLock, dagReader, pinManager, pinRemote }) {
const addAll = createAddAll({ gcLock, dagReader, pinManager })
this.addAll = addAll
this.add = createAdd({ addAll })
const rmAll = createRmAll({ gcLock, dagReader, pinManager })
this.rmAll = rmAll
this.rm = createRm({ rmAll })
this.ls = createLs({ dagReader, pinManager })
this.remote = pinRemote
}
}
module.exports = PinAPI
@@ -32,4 +34,5 @@ module.exports = PinAPI
* @typedef {import('..').PinManager} PinManager
* @typedef {import('..').AbortOptions} AbortOptions
* @typedef {import('..').CID} CID
* @typedef {import('./remote')} PinRemoteAPI
*/
@@ -0,0 +1,27 @@
'use strict'

const withTimeoutOption = require('ipfs-core-utils/src/with-timeout-option')

module.exports = ({ serviceRegistry }) => {
/**
* Asks a remote pinning service to pin an IPFS object from a given path
*
* @param {string|CID} cid
* @param {AddOptions & AbortOptions} options
* @returns {Promise<Pin>}
*/
async function add (cid, options) {
const { service } = options
const svc = serviceRegistry.serviceNamed(service)
Copy link
Contributor

@Gozala Gozala Apr 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like this will not handle a case when service was aded in one session and pin is added in the next one, because clients aren't populated unless add / ls / rm is called.

return svc.add(cid, options)
}

return withTimeoutOption(add)
}

/**
* @typedef {import('cids')} CID
* @typedef {import('ipfs-core-types/src/basic').AbortOptions} AbortOptions
* @typedef {import('ipfs-core-types/src/pin/remote').Pin} Pin
* @typedef {import('ipfs-core-types/src/pin/remote').AddOptions} AddOptions
*/