v1.0.8

IPFS Cluster v1.0.8 is a maintenance release.

This release updates dependencies (latest boxo and libp2p) and should bring a couple of Pebble-related improvements:

• We have upgraded Pebble's version. Some users have reported deadlocks in writes to Pebble (ipfs/ipfs-cluster#2009) and this seems to have helped.
• Pebble now supports 32-bit so it can be the default for all archs.
• We added a warning when Pebble's newest MajorFormatVersion is higher than
  what is used in the configuration. Users should increase their major_format_version
  to maintain forward-compatibility with future versions of Pebble.

Additionally, some bugs have been fixed and a couple of useful features added, as mentioned below.

List of changes

Breaking changes

There are no breaking changes on this release.

Features

• ipfshttp: support talking to Kubo over unix sockets | ipfs/ipfs-cluster#1999
• ipfsproxy: support talking to Kubo over unix sockets | ipfs/ipfs-cluster#2027
• pebble: enable in all archs as default datastore | ipfs/ipfs-cluster#2005 | ipfs/ipfs-cluster#2007
• pebble: set default MajorVersionFormat to newest | ipfs/ipfs-cluster#2019
• cluster: Announce and NoAnnounce options | ipfs/ipfs-cluster#952 | ipfs/ipfs-cluster#2010

Bug fixes

• ipfs-cluster-follow: issue numpin and pinqueue metrics to other peers | ipfs/ipfs-cluster#2011 | ipfs/ipfs-cluster#2016
• ipfshttp: do no pre-resolve node_multiaddresses | ipfs/ipfs-cluster#2004 | ipfs/ipfs-cluster#2017
• ipfsproxy: do no pre-resolve node_multiaddresses | ipfs/ipfs-cluster#2027
• pebble: deadlock | ipfs/ipfs-cluster#2009

Other changes

• The Dockerfile-bundle file has been removed (unmaintained) | ipfs/ipfs-cluster#1986
• Dependency upgrades | ipfs/ipfs-cluster#2007 | ipfs/ipfs-cluster#2018 | ipfs/ipfs-cluster#2026

Upgrading notices

Configuration changes

Two new options have been added to forcefully control the cluster peer libp2p host address announcements: cluster.announce_multiaddress and cluster.no_announce_multiaddress. Both take a slice of multiaddresses.

REST API

No changes.

Pinning Service API

No changes.

IPFS Proxy API

No changes.

Go APIs

No relevant changes.

Other

Nothing.

v1.0.7

IPFS Cluster v1.0.7 is a maintenance release.

This release updates dependencies and switches to the Boxo library suite with
the latest libp2p release.

See the notes below for a list of changes and bug fixes.

List of changes

Breaking changes

There are no breaking changes on this release.

Features

• API: Add a /health endpoint that bypasses authorization | ipfs/ipfs-cluster#1884 | ipfs/ipfs-cluster#1919
• Systemd notify support | ipfs/ipfs-cluster#1144 | ipfs/ipfs-cluster#1894
• Add support for pinning only on "untrusted" peers | ipfs/ipfs-cluster#1976 | ipfs/ipfs-cluster#1977
• Docker image with multiarch support | ipfs/ipfs-cluster#1085 | ipfs/ipfs-cluster#1984 | ipfs/ipfs-cluster#1368

Bug fixes

• MaxConcurrentCompactions missing from Pebble configuration | ipfs/ipfs-cluster#1895 | ipfs/ipfs-cluster#1900
• Missing newline in JSON stream from proxy API when doing pin/ls | ipfs/ipfs-cluster#1885 | ipfs/ipfs-cluster#1893

Other changes

• Dependency updates and upgrade to Boxo | ipfs/ipfs-cluster#1901 | ipfs/ipfs-cluster#1980

Upgrading notices

Configuration changes

A new option cluster.pin_only_on_untrusted_peers has been added, opposite to the pin_only_on_trusted_peers that already existed. Defaults to false. Both options cannot be true. When enabled, only "untrusted" peers are considered for pin allocations.

REST API

A new /health endpoint has been added, returns 204 (No Content) and no
body. It can be used to monitor that the service is running.

Pinning Service API

A new /health endpoint has been added, returns 204 (No Content) and no
body. It can be used to monitor that the service is running.

IPFS Proxy API

Calling /api/v0/pin/ls on the proxy api now adds a final new line at the end
of the response. This should align with what Kubo does.

Go APIs

No relevant changes.

Other

ipfs-cluster-service now sends a notification to systemd when it becomes
"ready" (that is, after all initialization is completed). This means systemd
service files for ipfs-cluster-service can use Type=notify.

The official docker images are now built with support for linux/amd64,
linux/arm/v7 and linux/arm64/v8 architectures. We have also switched to Alpine
Linux as base image (instead of Busybox). Binaries are now built with
CGO_ENABLED=0.

v1.0.6

IPFS Cluster v1.0.6 is a maintenance release with some small fixes. The main
change in this release is that pebble becomes the default datastore backend,
as we mentioned in the last release.

Pebble is the datastore backend used by CockroachDB and is inspired in
RocksDB. Upon testing, Pebble has demonstrated good performance and optimal
disk usage. Pebble incorporates modern datastore-backend features such as
compression, caching and bloom filters. Pebble is actively maintained by the
CockroachDB team and therefore seems like the best default choice for IPFS
Cluster.

Badger3, a very good alternative choice, becomes the new default for platforms
not supported by Pebble (mainly 32bit architectures). Badger and LevelDB are
still supported, but we heavily discourage their usage for new Cluster peers.

List of changes

Breaking changes

There are no breaking changes on this release.

Features

• Pebble: make the new default | ipfs/ipfs-cluster#1881 | ipfs/ipfs-cluster#1847
• ctl: support ipfs-cluster-ctl add --no-pin flag | ipfs/ipfs-cluster#1852

Bug fixes

• PinStatus information not up-to-date for re-pinned items with ongoing operations | ipfs/ipfs-cluster#1785 | ipfs/ipfs-cluster#1876
• Broken builds for 32-bit architectures | ipfs/ipfs-cluster#1884 | ipfs/ipfs-cluster#1854 | ipfs/ipfs-cluster#1851

Other changes

• Switch to ipfs/Kubo docker image | ipfs/ipfs-cluster#1846
• Dependency updates | ipfs/ipfs-cluster#1855 | ipfs/ipfs-cluster#1880
• Fix typo | ipfs/ipfs-cluster#1882

Upgrading notices

Configuration changes

The pebble section of the configuration has some additional options and new, adjusted defaults:

• pebble:

    "pebble": {
      "pebble_options": {
        "cache_size_bytes": 1073741824,
        "bytes_per_sync": 1048576,
        "disable_wal": false,
        "flush_delay_delete_range": 0,
        "flush_delay_range_key": 0,
        "flush_split_bytes": 4194304,
        "format_major_version": 1,
        "l0_compaction_file_threshold": 750,
        "l0_compaction_threshold": 4,
        "l0_stop_writes_threshold": 12,
        "l_base_max_bytes": 134217728,
        "levels": [
          {
            "block_restart_interval": 16,
            "block_size": 4096,
            "block_size_threshold": 90,
            "compression": 2,
            "filter_type": 0,
            "filter_policy": 10,
            "index_block_size": 4096,
            "target_file_size": 4194304
          },
          {
            "block_restart_interval": 16,
            "block_size": 4096,
            "block_size_threshold": 90,
            "compression": 2,
            "filter_type": 0,
            "filter_policy": 10,
            "index_block_size": 4096,
            "target_file_size": 8388608
          },
          {
            "block_restart_interval": 16,
            "block_size": 4096,
            "block_size_threshold": 90,
            "compression": 2,
            "filter_type": 0,
            "filter_policy": 10,
            "index_block_size": 4096,
            "target_file_size": 16777216
          },
          {
            "block_restart_interval": 16,
            "block_size": 4096,
            "block_size_threshold": 90,
            "compression": 2,
            "filter_type": 0,
            "filter_policy": 10,
            "index_block_size": 4096,
            "target_file_size": 33554432
          },
          {
            "block_restart_interval": 16,
            "block_size": 4096,
            "block_size_threshold": 90,
            "compression": 2,
            "filter_type": 0,
            "filter_policy": 10,
            "index_block_size": 4096,
            "target_file_size": 67108864
          },
          {
            "block_restart_interval": 16,
            "block_size": 4096,
            "block_size_threshold": 90,
            "compression": 2,
            "filter_type": 0,
            "filter_policy": 10,
            "index_block_size": 4096,
            "target_file_size": 134217728
          },
          {
            "block_restart_interval": 16,
            "block_size": 4096,
            "block_size_threshold": 90,
            "compression": 2,
            "filter_type": 0,
            "filter_policy": 10,
            "index_block_size": 4096,
            "target_file_size": 268435456
          }
        ],
        "max_open_files": 1000,
        "mem_table_size": 67108864,
        "mem_table_stop_writes_threshold": 20,
        "read_only": false,
        "wal_bytes_per_sync": 0
      }
    }

REST API

No changes.

Pinning Service API

No changes.

IPFS Proxy API

No changes.

Go APIs

No relevant changes.

Other

The --datastore flag to ipfs-cluster-service init now defaults to pebble
in most platforms, and to badger3 in those where Pebble is not supported
(arm, 386).

v1.0.5

IPFS Cluster v1.0.5 is a maintenance release with one main feature: support
for badger3 and pebble datastores.

Additionally, this release fixes compatibility with Kubo v0.18.0 and addresses
the crashes related to libp2p autorelay that affected the previous version.

pebble and badger3 are much newer backends that the already available
Badger and LevelDB. They are faster, use significantly less disk-space and
support additional options like compression. We have set pebble as the
default datastore used by the official Docker container, and we will be likely
making it the final default choice for new installations. In the meantime, we
encourage the community to try them out and provide feedback.

List of changes

Breaking changes

There are no breaking changes on this release.

Features

• Support for pebble and badger3 datastores | ipfs/ipfs-cluster#1809

Bug fixes

• Fix panic on boot (related to libp2p AutoRelay) | ipfs/ipfs-cluster#1796 | ipfs/ipfs-cluster#1797
• Fix error parsing quic-v1 multiaddresses from Kubo 0.18.0 | ipfs/ipfs-cluster#1835 | ipfs/ipfs-cluster#1841
• Fix error building docker container with newer git | ipfs/ipfs-cluster#1836
• Fix building master due to pinned go-mfs dependency | ipfs/ipfs-cluster#1817 | ipfs/ipfs-cluster#1818

Other changes

• Dependency updates (including libp2p v0.24.2) | ipfs/ipfs-cluster#1813 | ipfs/ipfs-cluster#1840
• Switch Docker containers to pebble by default | ipfs/ipfs-cluster#1842

Upgrading notices

Configuration changes

The datastore section of the configuration now supports the two new datastore backends:

• badger3:

    "badger3": {
      "gc_discard_ratio": 0.2,
      "gc_interval": "15m0s",
      "gc_sleep": "10s",
      "badger_options": {
        "dir": "",
        "value_dir": "",
        "sync_writes": false,
        "num_versions_to_keep": 1,
        "read_only": false,
        "compression": 0,
        "in_memory": false,
        "metrics_enabled": true,
        "num_goroutines": 8,
        "mem_table_size": 67108864,
        "base_table_size": 2097152,
        "base_level_size": 10485760,
        "level_size_multiplier": 10,
        "table_size_multiplier": 2,
        "max_levels": 7,
        "v_log_percentile": 0,
        "value_threshold": 100,
        "num_memtables": 5,
        "block_size": 4096,
        "bloom_false_positive": 0.01,
        "block_cache_size": 0,
        "index_cache_size": 0,
        "num_level_zero_tables": 5,
        "num_level_zero_tables_stall": 15,
        "value_log_file_size": 1073741823,
        "value_log_max_entries": 1000000,
        "num_compactors": 4,
        "compact_l_0_on_close": false,
        "lmax_compaction": false,
        "zstd_compression_level": 1,
        "verify_value_checksum": false,
        "checksum_verification_mode": 0,
        "detect_conflicts": false,
        "namespace_offset": -1
      }
    }

• pebble:

    "pebble": {
      "pebble_options": {
        "bytes_per_sync": 524288,
        "disable_wal": false,
        "flush_delay_delete_range": 0,
        "flush_delay_range_key": 0,
        "flush_split_bytes": 4194304,
        "format_major_version": 1,
        "l0_compaction_file_threshold": 500,
        "l0_compaction_threshold": 4,
        "l0_stop_writes_threshold": 12,
        "l_base_max_bytes": 67108864,
        "levels": [
          {
            "block_restart_interval": 16,
            "block_size": 4096,
            "block_size_threshold": 90,
            "compression": 1,
            "filter_type": 0,
            "index_block_size": 4096,
            "target_file_size": 2097152
          }
        ],
        "max_open_files": 1000,
        "mem_table_size": 4194304,
        "mem_table_stop_writes_threshold": 2,
        "read_only": false,
        "wal_bytes_per_sync": 0
      }
    }

In order to choose the backend during initialization, use the --datastore flag in ipfs-cluster-service init --datastore <backend>.

REST API

No changes.

Pinning Service API

No changes.

IPFS Proxy API

No changes.

Go APIs

No relevant changes.

Other

Docker containers now use pebble as the default datastore backend.
Nothing.

v1.0.4

IPFS Cluster v1.0.4 is a maintenance release addressing a couple of bugs and
adding more "state crdt" commands.

One of the bugs has potential to cause a panic, while a second one can
potentially dead-lock pinning operations and hang new pinning requests. We
recommend all users to upgrade as soon as possible.

List of changes

Breaking changes

There are no breaking changes on this release.

Features

• service: add "state crdt info/mark-dirty/mark-clean" commands | ipfs/ipfs-cluster#1771

Bug fixes

• Fix panic returned when request is canceled while rate-limited | ipfs/ipfs-cluster#1770
• Fix operationtracker not respecting context cancellations and never returning locks | ipfs/ipfs-cluster#1768
• Fix repinning of a CID not re-allocating errored peers as expected | ipfs/ipfs-cluster#1774

Other changes

No other changes.

Upgrading notices

Configuration changes

There are no configuration changes for this release.

REST API

No changes.

Pinning Service API

No changes.

IPFS Proxy API

No changes.

Go APIs

No relevant changes.

Other

Nothing.

v1.0.3

IPFS Cluster v1.0.3 is a maintenance release addressing some bugs and bringing
some improvements to error handling behavior, as well as a couple of small
features.

This release upgrades to the latest libp2p release (v0.22.0).

List of changes

Breaking changes

There are no breaking changes on this release.

Features

• ipfs proxy api: intercept and trigger cluster-pins on /block/put and /dag/put requests | ipfs/ipfs-cluster#1738 | ipfs/ipfs-cluster#1756
• "ipfs-cluster-service state crdt dot" command generates a DOT file from the CRDT DAG | ipfs/ipfs-cluster#1516

Bug fixes

• Fix leaking goroutines on aborted /add requests | ipfs/ipfs-cluster#1732
• Fix ARM build panics because of unaligned structs | ipfs/ipfs-cluster#1735 | ipfs/ipfs-cluster#1736 | ipfs/ipfs-cluster#1754
• Fix "blockPut response CID XXX does not match the multihash of any blocks sent" warning wrongly displayed | ipfs/ipfs-cluster#1706 | ipfs/ipfs-cluster#1755
• Fix behavior and error handling when IPFS is unavailable | ipfs/ipfs-cluster#1733 | ipfs/ipfs-cluster#1762

Other changes

• Dependency upgrades | ipfs/ipfs-cluster#1755
• Update hashicorp raft libraries | ipfs/ipfs-cluster#1757

Upgrading notices

Configuration changes

There are no configuration changes for this release.

REST API

No changes.

Pinning Service API

No changes.

IPFS Proxy API

The IPFS Proxy now intercepts /block/put and /dag/put requests. This happens as follows:

• The request is first forwarded "as is" to the underlying IPFS daemon, with
  the ?pin query parameter always set to false.
• If ?pin=true was set, a cluster pin is triggered for every block and dag
  object uploaded (reminder that these endpoints accept multipart uploads).
• Regular IPFS response to the uploads is streamed back to the user.

Go APIs

No relevant changes.

Other

Note that more than 10 failed requests to IPFS will now result in a rate-limit
of 1req/s for any request to IPFS. This may cause things to queue up instead
hammering the ipfs daemon with requets that fail. The rate limit is removed as
soon as one request succeeds.

Also note that now Cluster peers that are started will not become fully
operable until IPFS has been detected to be available: no metrics will be
sent, no recover operations will be run etc. essentially the Cluster peer will
wait for IPFS to be available before starting to do things that need IPFS to
be available, rather than doing them right away and have failures.

v1.0.2

IPFS Cluster v1.0.2 is a maintenance release with bug fixes and another iteration of the experimental support for the Pinning Services API that was introduced on v1.0.0, including Bearer token authorization support for both the REST and the Pinning Service APIs.

This release includes a security fix in the go-car library. The security issue allows an attacker to crash a cluster peer or cause excessive memory usage when uploading CAR files via the REST API (POST /add?format=car endpoint).

This also the first release after moving the project from the "ipfs" to the the "ipfs-cluster" Github organization, which means the project Go modules have new paths (everything is redirected though). The Docker builds remain inside the "ipfs" namespace (i.e. docker pull ipfs/ipfs-cluster).

IPFS Cluster is also ready to work with go-ipfs v0.13.0+. We recommend to upgrade.

List of changes

Breaking changes

Features

• REST/PinSVC API: support JWT bearer token authorization | #1703
• crdt: commit pending batched pins on shutdown | #1697 | 1719
• Export a prometheus - metric with the current disk informer value | #1725

Bug fixes

• Fix adding large directories | #1691 | #1700
• PinSVC API: fix compliance errors and bugs | #1704
• Pintracker: fix missing and wrong values in PinStatus object fields for recovered operations | #1705
• ctl: fix "Exp" label showing the pin timestamp instead of the experiation date | #1666 | #1716
• Pintracker: fix races causing wrong counts in metrics | #1717 | #1729
• Update go-car to v0.4.0 (security fixes) | #1730

Other changes

• Improve language, fix typos to changelog | #1667
• Update comment in docker-compose | #1689
• Migrate from ipfs/ipfs-cluster to ipfs-cluster/ipfs-cluster | #1694
• Enable spell-checking and fix spelling errors (US locale) | #1695
• Enable CodeQL analysis and fix security warnings | #1696
• Dependency upgrades: libp2p-0.20.1 etc. | #1711 | #1712 | #1724
• API: improve debug logging during tls setup | #1715

Upgrading notices

Configuration changes

There are no configuration changes for this release.

REST API

The REST API has a new POST /token endpoint, which returns a JSON object with a JWT token (when correctly authenticated).

This token can be used to authenticate using Authorization: Bearer header on subsequent requests.

The token is tied and verified against a basic authentication user and password, as configured in the basic_auth_credentials field.

At the moment we do not support revocation, expiration and other token options.

Pinning Service API

• The Pinning Service API has a new POST /token endpoint, which returns a JSON object with a JWT token (when correctly authenticated). See the REST API section above.

IPFS Proxy API

No changes to IPFS Proxy API.

Go APIs

• All cluster modules have new paths: every instance of "ipfs/ipfs-cluster" should now be "ipfs-cluster/ipfs-cluster".

Other

• go-ipfs v0.13.0 introduced some changes to the Block/Put API. IPFS Cluster now uses the cid-format option when performing Block-Puts. We believe the change does not affect adding blocks and that it should still work with previous go-ipfs versions, yet we recommend upgrading to go-ipfs v0.13.1 or later.

Prebuild Binaries

All prebuild binaries are available on dist.ipfs.io

v1.0.1

IPFS Cluster v1.0.1 is a maintenance release ironing out some issues and
bringing a couple of improvements around observability of cluster performance:

• We have fixed the ipfscluster_pins metric and added a few new ones that
  help determine how fast the cluster can pin and add blocks.
• We have added a new Informer that broadcasts current pinning-queue size,
  which means we can take this information into account when making
  allocations, essentially allowing peers with big pinning queues to be
  relieved by peers with smaller pinning queues.

Please read below for a list of changes and things to watch out for.

List of changes

Breaking changes

Peers running IPFS Cluster v1.0.0 will not be able to read the pin's user-set
metadata fields for pins submitted by peers in later versions, since metadata
is now stored on a different protobuf field. If this is an issue, all peers in
the cluster should upgrade.

Features

• Pinqueue Informer: let pinning queue size inform allocation selection | ipfs/ipfs-cluster#1649 | ipfs/ipfs-cluster#1657
• Metrics: add additional Prometheus metrics | ipfs/ipfs-cluster#1650 | ipfs/ipfs-cluster#1659

Bug fixes

• Fix: state import can result in different CRDT-heads | ipfs/ipfs-cluster#1547 | ipfs/ipfs-cluster#1664
• Fix: ipfs-cluster-ctl pin ls hangs | ipfs/ipfs-cluster#1663
• Fix: restapi client panics on retry | ipfs/ipfs-cluster#1655 | ipfs/ipfs-cluster#1662
• Fix: bad behavior while adding and ipfs is down | ipfs/ipfs-cluster#1646
• Fix: ipfscluster_pins metric issues bad values | ipfs/ipfs-cluster#1645

Other changes

• Dependency upgrades (includes go-libp2p v0.19.1) | ipfs/ipfs-cluster#1660
• Build with go1.18 | ipfs/ipfs-cluster#1661
• Do not issue freespace metrics when freespace is 0 | ipfs/ipfs-cluster#1656
• Convert pinning/queued/error metrics go gauges | ipfs/ipfs-cluster#1647 | ipfs/ipfs-cluster#1651

Upgrading notices

Configuration changes

There is a new pinqueue configuration object inside the informer section on newly initialized configurations:

  "informer": {
    ...
    "pinqueue": {
      "metric_ttl": "30s",
      "weight_bucket_size": 100000
    },
	...

This enables the Pinqueue Informer, which broadcasts metrics containing the size of the pinqueue with the metric weight divided by weight_bucket_size. The new metric is not used for allocations by default, and it needs to be manually added to the allocate_by option in the allocator, usually like:

"allocator": {
   "balanced": {
     "allocate_by": [
       "tag:group",
       "pinqueue",
       "freespace"
     ]
   }

REST API

No changes to REST API.

IPFS Proxy API

No changes to IPFS Proxy API.

Go APIs

No relevant changes to Go APIs, other than the PinTracker interface now requiring a PinQueueSize method.

Other

The following metrics are now available in the Prometheus endpoint when enabled:

ipfscluster_pins_ipfs_pins gauge
ipfscluster_pins_pin_add counter
ipfscluster_pins_pin_add_errors counter
ipfscluster_blocks_put counter
ipfscluster_blocks_added_size counter
ipfscluster_blocks_added counter
ipfscluster_blocks_put_error counter

The following metrics were converted from counter to gauge:

ipfscluster_pins_pin_queued
ipfscluster_pins_pinning
ipfscluster_pins_pin_error

Peers that are reporting freespace as 0 and which use this metric to
allocate pins, will no longer be available for allocations (they stop
broadcasting this metric). This means setting StorageMax on IPFS to 0
effectively prevents any pins from being explicitly allocated to a peer
(that is, when replication_factor != everywhere).

v1.0.0

IPFS Cluster v1.0.0 is a major release that represents that this project has reached maturity and is able to perform and scale on production environment (50+ million pins and 20 nodes).

This is a breaking release, v1.0.0 cluster peers are not compatible with previous cluster peers as we have bumped the RPC protocol version (which had remained unchanged since 0.12.0).

For a full list of changes, see the CHANGELOG.

v0.14.5

See the CHANGELOG.