Skip to content
Permalink
Browse files

merge with ipfs cluster master (#5)

* Add the build tools download gateway.

Signed-off-by: Yi Wang <wangyi8848@gmail.com>

* add hive.cluster mark.
due to this change, hive.cluster will be isolated from typical ipfs.cluster.

Signed-off-by: Yi Wang <wangyi@storswift.com>

* Fix HTTPs with DNS multiaddresses

Before we resolved all /dns*/ multiaddresses before we used them.

When using HTTPs, the Go HTTP Client only sees the resolved IP address
and it is unable to negotiate TLS with a cerficate because the request
is not going to the hostname the certificate is signed for, but to
the IP. This leverages a recent feature in go-multiaddr-net
and uses directly the user-provided hostname.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Fix interpreting Host parameter correctly.

We should deprecate passing in Host/Port in the config,
but in the meantime, it hardcoded /dns4/, meaning that if
someone placed an ipv6 address in there things would break badly
and weirdly.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Added tests for /monitor/metrics/{metrics_type}

Added API and client tests for GET /monitor/metrics/{metrics_type}

Fixes ipfs#587

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

* We are using https://github.com/chriscool/sharness

We aren't using https://github.com/mlafeldt/sharness, code reference
below
https://github.com/ipfs/ipfs-cluster/blob/fdfe8def9467893d451e1fcb8ea3fb980c8c1389/Makefile#L67

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

* Sharness tests for ipfs-cluster-ctl health metrics

Added sharness tests for `ipfs-cluster-ctl health metrics <metricname>`

Fixes ipfs#587

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

* Added tests for /monitor/metrics/{metrics_type}

Move ctl-health sharness tests to apprpriate file

Since the API is using the RPC mock to request metrics and it always
returns a mocked test metric we might just do c.Metrics("somemetricstype")
and check that there is no error. Here we just want to check that the
client is hitting an API endpoint (and understands the response).

Fixes ipfs#587

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

* Fix ipfs#632: Handle "stream-channels" in /add endpoints

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Fix $632: Test stream-channels=false in /add endpoint

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Fix ipfs#632: Apply -Q option to --enc=json when adding.

This will only print the latest JSON update when adding
with -Q.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Fix ipfs#445: Implemented status filter for ipfs-cluster-ctl

License: MIT
Signed-off-by: Paul Jewell <sona1111@zoho.com>

* Fix ipfs#445: Reduced length of filter help string

License: MIT
Signed-off-by: Paul Jewell <sona1111@zoho.com>

* Fix ipfs#445:

-Fixed logic issue in match condition of 'filterStatus' function
-Added and verified success of test provided by @lanzafame
-Attempted to condense code and apply other cleanup provided by @lanzafame

License: MIT
Signed-off-by: Paul Jewell <sona1111@zoho.com>

* Fix ipfs#445:

-Changed some 'snake case' to 'camel case' in accordance with code climate

License: MIT
Signed-off-by: Paul Jewell <sona1111@zoho.com>

* Status filters for `ipfs-cluster-ctl status`

Added filter option to `ipfs-cluster-ctl status`

When the --filter is passed, it will only fetch the peer information
where status of the pin matches with the filter value.
Valid filter values are tracker status types(i.e., "pinned",
"pin_error", "unpinning" etc), an alias of tracker status type (i.e.,
"queued" or "error"), comma separated list of tracker status type
and/or it aliases(i.e., "error,pinning")

On passing invalid filter value no status information will be shown

In particular, the filter would remove elements from []GlobalPinInfo
when none of the peers in GlobalPinInfo match the filter. If one peer
in the GlobalPinInfo matches the filter, the whole object is returned,
including the information for the other peers which may or not match it.

filter option works on statusAll("GET /pins"). For fetching pin status
for a CID("GET /pins/<cid>"), filter option would have no effect

Fixes ipfs#445

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

* Status filters for `ipfs-cluster-ctl status`

Added clients tests

Fixes ipfs#445

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

* Status filters for `ipfs-cluster-ctl status`

Added a fail case where an invalid filter is passed in.
Update `api.rpcClient.Call` to `api.rpcClient.CallContext` and pass
in the Request context r.Context() so that context can be cancelled
when the request is cancelled by caller

Fixes ipfs#445

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

* Status filters for `ipfs-cluster-ctl status`

Passing `make check`

Fixes ipfs#445

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

* Status filters for `ipfs-cluster-ctl status`

Improved matching of filters and tracker status

Fixes ipfs#445

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

* Wrap help message for less than 120 characters

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

* Status filters for `ipfs-cluster-ctl status`

Optimized filter to tracker status matching by using bitwise
comparisions

License: MIT
Signed-off-by: Kishan Mohanbhai Sagathiya <kishansagathiya@gmail.com>

* Fix ipfs#632: Make sure StreamChannels is enabled in rest/client.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Feat ipfs#632: Keep default /add behaviour outside of conditional block

Per review comment.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Feat ipfs#445: Use TrackerStatus as filter. Simplify and small misc.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Feat ipfs#445: Clarify about 0 filter value.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* add uid register and uid name changing API.

Signed-off-by: Yi Wang <wangyi@storswift.com>

* add hive api: UidNew, UidLogIn, FilesCp, FilesFlush, FilesLs

Signed-off-by: Yi Wang <wangyi@storswift.com>

* Feat ipfs#445: Catch invalid filter strings in ipfs-cluster-ctl.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Issue ipfs#445: Fix test

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Feat ipfs#445: Fix string test with TrackerStatus == 0

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* add FilesMkdir, FilesMv, and FilesRm API.

Signed-off-by: Yi Wang <wangyi@storswift.com>

* Fix ipfs#632: Add --no-stream to ipfs-cluster-ctl

tl;dr: this solves the user's immediate need and, even if not tne strictest
solution, it is the simplest one.

I think we should not have the server buffer output when we can do it rather
easily client side and have the clients use their own memory for the task even
if `stream-channels=false` would do this.

We can always change the approach, but this is the minimal solution to
json array with all the AddedOutput things.

We might not buffer at all and hack a `[`, `,` separating elements and `]`
at the end, when json encoding is enabled, etc. But that would not be clean,
specially if we mean to support more output formats at some point.

Enabling supporting stream-channels=false in the api/rest/client means adding
new methods, with tests, modifying the interface etc etc. for what is
essentially a presentation issue in "ctl" in the end. Similarly we could
buffer inside the client, but it is so trivial that I don't see advatange.
We should also start thinking about moving to streaming API endpoints and
when that moment arrives we shall revisit this discussion.

I have removed the hacky manual output parts by declaring a custom
addedOutputQuiet type which wraps added output and is understood by
the formatters.go helpers.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* add FilesRead, FilesStat, FilesWrite.

Known bug: FilesWrite may occur writing interrupt.

Signed-off-by: Yi Wang <wangyi@storswift.com>

* Issue ipfs#632: Fix wrong name for flag. Address review comments

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Fix ipfs#382 (again): A better strategy for handling proxy headers

This changes the current strategy to extract headers from the IPFS daemon to
use them for hijacked endpoints in the proxy. The ipfs daemon is a bit of a
mess and what we were doing is not really reliable, specially when it comes to
setting CORS headers right (which we were not doing).

The new approach is:

* For every hijacked request, make an OPTIONS request to the same path, with
the given Origin, to the IPFS daemon and extract some CORS headers from
that. Use those in the hijacked response

* Avoid hijacking OPTIONS request, they should always go through so the IPFS
daemon controls all the CORS-preflight things as it wants.

* Similar to before, have a only-once-triggered request to extract other
interesting or custom headers from a fixed IPFS endpoint.  This allows us to
have the proxy forward other custom headers and to catch
`Access-Control-Expose-Methods`. The difference is that the endpoint use for
this and the additional headers are configurable by the user (but with hidden
configuration options because this is quite exotic from regular usage).

Now the implementation:

* Replaced the standard Muxer with gorilla/mux (I have also taken the change
to update the gxed version to the latest tag). This gives us much better
matching control over routes and allows us to not handle OPTIONS requests.

* This allows also to remove the extractArgument code and have proper handlers
for the endpoints passing command arguments as the last segment of the URL. A
very simple handler that wraps the default ones can be used to extract the
argument from the url and put it in the query.  Overall much cleaner this way.

* No longer capture interesting headers from any random proxied request.  This
made things complicated with a wrapping handler. We will just trigger the one
request to do it when we need it.

* When preparing the headers for the hijacked responses:
  * Trigger the OPTIONS request and figure out which CORS things we should set
  * Set the additional headers (perhaps triggering a POST request to fetch them)
  * Set our own headers.

* Moved all the headers stuff to a new headers.go file.

* Added configuration options (hidden by default) to:
  * Customize the extract headers endpoint
  * Customize what additional headers are extracted
  * Use HTTPs when talking to the IPFS API
    * I haven't tested this, but I did not want to have hardcoded 'http://' urls
      around, as before.

* Added extra testing for this, and tested manually a lot comparing the
daemon original output with our hijacked endpoint outputs while looking
at the API traffic with ngrep and making sure the requets happen as expected.
Also tested with IPFS companion in FF and Chrome.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* ipfsproxy: add ExtractHeaderTTL option to config

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Fix ipfs#382: Add TTL for cached headers

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Fix typos in comments

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Fix ipfs#639: Import rs/cors with Gx

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Fix ipfs#639: Add CORS options to restapi

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Fix ipfs#639: restapi: Handle CORS preflight requests (OPTIONS)

This adds support for handling preflight requests in the REST API
and fixes currently mostly broken CORS.

Before we just let the user add custom response headers to the
configuration "headers" key but this is not the best way because
CORs headers and requests need special handling and doing it wrong
has security implications.

Therefore, I have added specific CORS-related configuration options
which control CORS behavour. We are forced to change the "headers"
defaults and will notify the users about this in the changelog.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* ipfsproxy: fix tests for new configuration keys

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* ipfsproxy: fix typos in comments

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* restapi: minor codeclimate issue

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Fix ipfs#639: Do not break start by complaining of unset CORSMaxAge

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Fix ipfs#639: Enforce basic auth for all requests when enabled

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Sharness: update configuration files used in sharness

Maintenance

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* config: Fix confusing errors

The JSON parsing of the config could error, but we skipped error checking and
use Validate() at the end. This caused that maybe some JSON parsing errors
where logged but the final error when validating the configuration came from
somewhere different, creating very confusing error messages for the user.

This changes this, along with removing hardcoded section lists. This also
removes a Sharder component section because, AFAIK, it was a left over
from the sharding branch and in the end there is no separate sharding
component that needs a config.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* sharness: Fix test typo causing an empty grep

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Config: Interpret empty duration strings as duration = 0.

In practice, we allowed this already, because parsing failed
but Validate() succeeded.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* config: do not handle "" durations (and do not error)

They should not be interpreted as 0, since that may overwrite
defaults which are not 0. We simply need to do nothing.

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Update sharness/t0032-ctl-health.sh

Fix the pid extraction in test.

Co-Authored-By: hsanjuan <hsanjuan@users.noreply.github.com>

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* sharness: test should check agains cluster peer ID

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Release 0.8.0-rc1

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* gx publish 0.8.0-rc1

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Changelog for 0.8.0 release

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Release 0.8.0

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* gx publish 0.8.0

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Fix changelog date

License: MIT
Signed-off-by: Hector Sanjuan <code@hector.link>

* Fix bug in filesWrite and add NamePublish.

Signed-off-by: Yi Wang <wangyi@storswift.com>

* add uid/info API

Signed-off-by: Yi Wang <wangyi@storswift.com>
  • Loading branch information...
yeewang authored and coder-lb committed Jan 21, 2019
1 parent 0411cbf commit 9506f548bd69c2637a5f3f0b57b82abbb06e6130
@@ -1 +1 @@
0.7.0: QmbGeAqG8wWUqBggxHRCP5ErJGMJHm8SihCfurMmxRU4za
0.8.0: QmfDMnXBd6ChZziataTVWszPFKo1Yr4AwMYWPHXKBfCUAe
@@ -1,10 +1,164 @@
# IPFS Cluster Changelog

### v0.8.0 - 2019-01-16

#### Summary

IPFS Cluster version 0.8.0 comes with a few useful features and some bugfixes.
A significant amount of work has been put to correctly handle CORS in both the
REST API and the IPFS Proxy endpoint, fixing some long-standing issues (we
hope once are for all).

There has also been heavy work under the hood to separate the IPFS HTTP
Connector (the HTTP client to the IPFS daemon) from the IPFS proxy, which is
essentially an additional Cluster API. Check the configuration changes section
below for more information about how this affects the configuration file.

Finally we have some useful small features:

* The `ipfs-cluster-ctl status --filter` option allows to just list those
items which are still `pinning` or `queued` or `error` etc. You can combine
multiple filters. This translates to a new `filter` query parameter in the
`/pins` API endpoint.
* The `stream-channels=false` query parameter for the `/add` endpoint will let
the API buffer the output when adding and return a valid JSON array once done,
making this API endpoint behave like a regular, non-streaming one.
`ipfs-cluster-ctl add --no-stream` acts similarly, but buffering on the client
side. Note that this will cause in-memory buffering of potentially very large
responses when the number of added files is very large, but should be
perfectly fine for regular usage.
* The `ipfs-cluster-ctl add --quieter` flag now applies to the JSON output
too, allowing the user to just get the last added entry JSON object when
adding a file, which is always the root hash.

#### List of changes

##### Features

* IPFS Proxy extraction to its own `API` component: `ipfsproxy` | [ipfs/ipfs-cluster#453](https://github.com/ipfs/ipfs-cluster/issues/453) | [ipfs/ipfs-cluster#576](https://github.com/ipfs/ipfs-cluster/issues/576) | [ipfs/ipfs-cluster#616](https://github.com/ipfs/ipfs-cluster/issues/616) | [ipfs/ipfs-cluster#617](https://github.com/ipfs/ipfs-cluster/issues/617)
* Add full CORS handling to `restapi` | [ipfs/ipfs-cluster#639](https://github.com/ipfs/ipfs-cluster/issues/639) | [ipfs/ipfs-cluster#640](https://github.com/ipfs/ipfs-cluster/issues/640)
* `restapi` configuration section entries can be overriden from environment variables | [ipfs/ipfs-cluster#609](https://github.com/ipfs/ipfs-cluster/issues/609)
* Update to `go-ipfs-files` 2.0 | [ipfs/ipfs-cluster#613](https://github.com/ipfs/ipfs-cluster/issues/613)
* Tests for the `/monitor/metrics` endpoint | [ipfs/ipfs-cluster#587](https://github.com/ipfs/ipfs-cluster/issues/587) | [ipfs/ipfs-cluster#622](https://github.com/ipfs/ipfs-cluster/issues/622)
* Support `stream-channels=fase` query parameter in `/add` | [ipfs/ipfs-cluster#632](https://github.com/ipfs/ipfs-cluster/issues/632) | [ipfs/ipfs-cluster#633](https://github.com/ipfs/ipfs-cluster/issues/633)
* Support server side `/pins` filtering | [ipfs/ipfs-cluster#445](https://github.com/ipfs/ipfs-cluster/issues/445) | [ipfs/ipfs-cluster#478](https://github.com/ipfs/ipfs-cluster/issues/478) | [ipfs/ipfs-cluster#627](https://github.com/ipfs/ipfs-cluster/issues/627)
* `ipfs-cluster-ctl add --no-stream` option | [ipfs/ipfs-cluster#632](https://github.com/ipfs/ipfs-cluster/issues/632) | [ipfs/ipfs-cluster#637](https://github.com/ipfs/ipfs-cluster/issues/637)
* Upgrade dependencies and libp2p to version 6.0.29 | [ipfs/ipfs-cluster#624](https://github.com/ipfs/ipfs-cluster/issues/624)

##### Bug fixes

* Respect IPFS daemon response headers on non-proxied calls | [ipfs/ipfs-cluster#382](https://github.com/ipfs/ipfs-cluster/issues/382) | [ipfs/ipfs-cluster#623](https://github.com/ipfs/ipfs-cluster/issues/623) | [ipfs/ipfs-cluster#638](https://github.com/ipfs/ipfs-cluster/issues/638)
* Fix `ipfs-cluster-ctl` usage with HTTPs and `/dns*` hostnames | [ipfs/ipfs-cluster#626](https://github.com/ipfs/ipfs-cluster/issues/626)
* Minor fixes in sharness | [ipfs/ipfs-cluster#641](https://github.com/ipfs/ipfs-cluster/issues/641) | [ipfs/ipfs-cluster#643](https://github.com/ipfs/ipfs-cluster/issues/643)
* Fix error handling when parsing the configuration | [ipfs/ipfs-cluster#642](https://github.com/ipfs/ipfs-cluster/issues/642)



#### Upgrading notices

This release comes with some configuration changes that are important to notice,
even though the peers will start with the same configurations as before.

##### Configuration changes

##### `ipfsproxy` section

This version introduces a separate `ipfsproxy` API component. This is
reflected in the `service.json` configuration, which now includes a new
`ipfsproxy` subsection under the `api` section. By default it looks like:

```js
"ipfsproxy": {
"node_multiaddress": "/ip4/127.0.0.1/tcp/5001",
"listen_multiaddress": "/ip4/127.0.0.1/tcp/9095",
"read_timeout": "0s",
"read_header_timeout": "5s",
"write_timeout": "0s",
"idle_timeout": "1m0s"
}
```

We have however added the necessary safeguards to keep backwards compatibility
for this release. If the `ipfsproxy` section is empty, it will be picked up from
the `ipfshttp` section as before. An ugly warning will be printed in this case.

Based on the above, the `ipfshttp` configuration section loses the
proxy-related options. Note that `node_multiaddress` stays in both component
configurations and should likely be the same in most cases, but you can now
potentially proxy requests to a different daemon than the one used by the
cluster peer.

Additional hidden configuration options to manage custom header extraction
from the IPFS daemon (for power users) have been added to the `ipfsproxy`
section but are not shown by default when initializing empty
configurations. See the documentation for more details.

###### `restapi` section

The introduction of proper CORS handling in the `restapi` component introduces
a number of new keys:

```js
"cors_allowed_origins": [
"*"
],
"cors_allowed_methods": [
"GET"
],
"cors_allowed_headers": [],
"cors_exposed_headers": [
"Content-Type",
"X-Stream-Output",
"X-Chunked-Output",
"X-Content-Length"
],
"cors_allow_credentials": true,
"cors_max_age": "0s"
```

Note that CORS will be essentially unconfigured when these keys are not
defined.

The `headers` key, which was used before to add some CORS related headers
manually, takes a new empty default. **We recommend emptying `headers` from
any CORS-related value.**


##### REST API

The REST API is fully backwards compatible:

* The `GET /pins` endpoint takes a new `?filter=<filter>` option. See
`ipfs-cluster-ctl status --help` for acceptable values.
* The `POST /add` endpoint accepts a new `?stream-channels=<true|false>`
option. By default it is set to `true`.

##### Go APIs

The signature for the `StatusAll` method in the REST `client` module has
changed to include a `filter` parameter.

There may have been other minimal changes to internal exported Go APIs, but
should not affect users.

##### Other

Proxy requests which are handled by the Cluster peer (`/pin/ls`, `/pin/add`,
`/pin/rm`, `/repo/stat` and `/add`) will now attempt to fully mimic ipfs
responses to the header level. This is done by triggering CORS pre-flight for
every hijacked request along with an occasional regular request to `/version`
to extract other headers (and possibly custom ones).

The practical result is that the proxy now behaves correctly when dropped
instead of IPFS into CORS-aware contexts (like the browser).

---

### v0.7.0 - 2018-11-01

#### Summary

IPFS version 0.7.0 is a maintenance release that includes a few bugfixes and some small features.
IPFS Cluster version 0.7.0 is a maintenance release that includes a few bugfixes and some small features.

Note that the REST API response format for the `/add` endpoint has changed. Thus all clients need to be upgraded to deal with the new format. The `rest/api/client` has been accordingly updated.

@@ -34,58 +34,103 @@ func AddMultipartHTTPHandler(
) (cid.Cid, error) {
var dags adder.ClusterDAGService
output := make(chan *api.AddedOutput, 200)
flusher, flush := w.(http.Flusher)

if params.Shard {
dags = sharding.New(rpc, params.PinOptions, output)
} else {
dags = local.New(rpc, params.PinOptions)
}

enc := json.NewEncoder(w)
if outputTransform == nil {
outputTransform = func(in *api.AddedOutput) interface{} { return in }
}

// This must be application/json otherwise go-ipfs client
// will break.
w.Header().Set("Content-Type", "application/json")
// Browsers should not cache when streaming content.
// Browsers should not cache these responses.
w.Header().Set("Cache-Control", "no-cache")
// Custom header which breaks js-ipfs-api if not set
// https://github.com/ipfs-shipyard/ipfs-companion/issues/600
w.Header().Set("X-Chunked-Output", "1")

// Used by go-ipfs to signal errors half-way through the stream.
w.Header().Set("Trailer", "X-Stream-Error")

// We need to ask the clients to close the connection
// (no keep-alive) of things break badly when adding.
// https://github.com/ipfs/go-ipfs-cmds/pull/116
w.Header().Set("Connection", "close")
w.WriteHeader(http.StatusOK)

if outputTransform == nil {
outputTransform = func(in *api.AddedOutput) interface{} { return in }
var wg sync.WaitGroup
if !params.StreamChannels {
// in this case we buffer responses in memory and
// return them as a valid JSON array.
wg.Add(1)
var bufOutput []interface{} // a slice of transformed AddedOutput
go func() {
defer wg.Done()
bufOutput = buildOutput(output, outputTransform)
}()

enc := json.NewEncoder(w)
add := adder.New(dags, params, output)
root, err := add.FromMultipart(ctx, reader)
if err != nil { // Send an error
logger.Error(err)
w.WriteHeader(http.StatusInternalServerError)
errorResp := api.Error{
Code: http.StatusInternalServerError,
Message: err.Error(),
}

if err := enc.Encode(errorResp); err != nil {
logger.Error(err)
}
wg.Wait()
return root, err
}
wg.Wait()
w.WriteHeader(http.StatusOK)
enc.Encode(bufOutput)
return root, err
}

var wg sync.WaitGroup
// handle stream-adding. This should be the default.

// https://github.com/ipfs-shipyard/ipfs-companion/issues/600
w.Header().Set("X-Chunked-Output", "1")
// Used by go-ipfs to signal errors half-way through the stream.
w.Header().Set("Trailer", "X-Stream-Error")
w.WriteHeader(http.StatusOK)
wg.Add(1)
go func() {
defer wg.Done()
for v := range output {
err := enc.Encode(outputTransform(v))
if err != nil {
logger.Error(err)
break
}
if flush {
flusher.Flush()
}
}
streamOutput(w, output, outputTransform)
}()

add := adder.New(dags, params, output)
root, err := add.FromMultipart(ctx, reader)
if err != nil {
logger.Error(err)
// Set trailer with error
w.Header().Set("X-Stream-Error", err.Error())
}
wg.Wait()
return root, err
}

func streamOutput(w http.ResponseWriter, output chan *api.AddedOutput, transform func(*api.AddedOutput) interface{}) {
flusher, flush := w.(http.Flusher)
enc := json.NewEncoder(w)
for v := range output {
err := enc.Encode(transform(v))
if err != nil {
logger.Error(err)
break
}
if flush {
flusher.Flush()
}
}
}

func buildOutput(output chan *api.AddedOutput, transform func(*api.AddedOutput) interface{}) []interface{} {
var finalOutput []interface{}
for v := range output {
finalOutput = append(finalOutput, transform(v))
}
return finalOutput
}
@@ -24,31 +24,33 @@ type AddedOutput struct {
type AddParams struct {
PinOptions

Recursive bool
Layout string
Chunker string
RawLeaves bool
Hidden bool
Wrap bool
Shard bool
Progress bool
CidVersion int
HashFun string
Recursive bool
Layout string
Chunker string
RawLeaves bool
Hidden bool
Wrap bool
Shard bool
Progress bool
CidVersion int
HashFun string
StreamChannels bool
}

// DefaultAddParams returns a AddParams object with standard defaults
func DefaultAddParams() *AddParams {
return &AddParams{
Recursive: false,
Layout: "", // corresponds to balanced layout
Chunker: "size-262144",
RawLeaves: false,
Hidden: false,
Wrap: false,
Shard: false,
Progress: false,
CidVersion: 0,
HashFun: "sha2-256",
Recursive: false,
Layout: "", // corresponds to balanced layout
Chunker: "size-262144",
RawLeaves: false,
Hidden: false,
Wrap: false,
Shard: false,
Progress: false,
CidVersion: 0,
HashFun: "sha2-256",
StreamChannels: true,
PinOptions: PinOptions{
ReplicationFactorMin: 0,
ReplicationFactorMax: 0,
@@ -90,7 +92,7 @@ func AddParamsFromQuery(query url.Values) (*AddParams, error) {
case "trickle", "balanced", "":
// nothing
default:
return nil, errors.New("parameter trickle invalid")
return nil, errors.New("layout parameter invalid")
}
params.Layout = layout

@@ -153,6 +155,11 @@ func AddParamsFromQuery(query url.Values) (*AddParams, error) {
params.ShardSize = shardSize
}

err = parseBoolParam(query, "stream-channels", &params.StreamChannels)
if err != nil {
return nil, err
}

return params, nil
}

@@ -173,6 +180,7 @@ func (p *AddParams) ToQueryString() string {
query.Set("progress", fmt.Sprintf("%t", p.Progress))
query.Set("cid-version", fmt.Sprintf("%d", p.CidVersion))
query.Set("hash", p.HashFun)
query.Set("stream-channels", fmt.Sprintf("%t", p.StreamChannels))
return query.Encode()
}

@@ -190,5 +198,6 @@ func (p *AddParams) Equals(p2 *AddParams) bool {
p.Hidden == p2.Hidden &&
p.Wrap == p2.Wrap &&
p.CidVersion == p2.CidVersion &&
p.HashFun == p2.HashFun
p.HashFun == p2.HashFun &&
p.StreamChannels == p2.StreamChannels
}

0 comments on commit 9506f54

Please sign in to comment.
You can’t perform that action at this time.