Skip to content
Dec 28, 2019
Bump RELEASE_NUMBER 32 => 33 for 1.9.3.0

@ldmberman ldmberman released this Dec 28, 2019 · 1 commit to master since this release

**Improvements & bug fixes**

The release includes a few improvements to transaction processing and gossiping mechanisms, including in the priority queue, plus miscellaneous bug fixes.

- A bug was fixed in the priority queue which caused it to distribute transactions too quickly.
- Improvements to the core gossiping mechanics of the network.
- Miscellaneous bug fixes.

**Upgrade instructions**

N.1.9.3.0 comes with the prebuilt binaries for the Linux and Darwin x86_64 platforms.

Download the corresponding archive and unpack it:

```
tar -xzf arweave-1.9.3.0.linux-x86_64.tar.gz
```

It makes sense to unpack it inside a dedicated directory. You can always move this directory around, but the miner may not work if you move only some of the files. The weave data would, by default, be stored in this directory, too, but it can be overridden using the data_dir command-line argument.
The archive contents look like this:

```
bin/  data/  erts-10.3/  lib/  releases/
```

To run the miner, execute ./bin/start <arguments>. See the mining guide for more details.
If your OS/platform architecture is not in the list, check README for how to build the miner from sources.

If you want to run the miner from the existing Git folder, execute the following command to upgrade:

```
git fetch --all --tags && git checkout -f N.1.9.3.0
```

You can now run the miner using the arweave-server script.

**Support**

For more details on how to run a mining  node, please see our mining guide - https://docs.arweave.org/info/mining/mining-guide. If you have any issues upgrading or would like to know more about the release, feel free to reach out to us in the #mining channel on Discord Arweave Dev Talk server (https://discord.gg/DjAFMJc) or email us at team@arweave.org.
Assets 4

@ldmberman ldmberman released this Dec 12, 2019 · 11 commits to master since this release

The release comes with the new build system. No need to install any dependencies anymore - simply download and run the binary! Please, read the upgrade upgrade instructions below for details - the upgrade instructions for the previous releases would not apply here.

Improvements and bugfixes, the miner:

  • The polling mode was made more efficient. In addition to that, even outside the polling mode the node will ask its trusted peers for blocks if it has not received any in the last minute.
  • The number of best peers to propagate transactions and blocks to can be specified in the command line.
  • The number of file descriptors to use for incoming connections can be specified on the command line. After the limit is reached, the requests are queued.
  • Collecting transactions during fork recovery now respects protocol limits to avoid abuse.
  • Requests with valid transactions are excluded from the IP-based throttling to ensure timely propagation of excessive transaction volumes.
  • Metrics for uploaded and downloaded bytes per endpoint were added.

Improvements and bugfixes, the gateway:

  • The SQLite driver was replaced, the previous one had an issue where it occasionally would fail to restart after a crash.
  • The missing CORS headers were added to the /:path and /:path/:subpath endpoints.
  • The number of file descriptors necessary for incoming gateway connections was reduced.
  • Metadata of all transactions of a block is inserted in a single database transaction now.

Upgrade instructions

N.1.9.2.0 comes with the prebuilt binaries for the Linux and Darwin x86_64 platforms.

Download the corresponding archive and unpack it.

tar -xzf arweave-1.9.2.0.linux-x86_64.tar.gz

It makes sense to unpack it inside a dedicated directory. You can always move this directory around, but the miner may not work if you move only some of the files. The weave data would, by default, be stored in this directory, too, but it can be overridden using the data_dir command-line argument.

The archive contents look like this:

bin/  data/  erts-10.3/  lib/  releases/

To run the miner, execute ./bin/start <arguments>. See the mining guide for more details.

If your OS/platform architecture is not in the list, check README for how to build the miner from sources.

If you have been already mining before this update, you can either download and unpack the archive in your Git arweave folder and run ./bin/start ... or execute the following to continue building the miner from sources:

git fetch --all --tags && git checkout -f N.1.9.2.0
git submodule update
rm -rf lib _build

You can now run the miner using the arweave-server script.

You can also create a new directory and unpack the archive there, but make sure you use the same data directory (data_dir argument) as you did before. If you used the default, it would be data_dir <path-to-your-previous-arweave-folder> in the mining command. Check this section of the guide if you would like to move your data elsewhere.

If you operate a gateway, note that the TLS certificates need to be put into the apps/arweave/priv/tls folder.

Support

For more details on how to run the node, please see our mining guide - https://docs.arweave.org/info/mining/mining-guide. If you have any issues upgrading or would like to know more about the release, feel free to reach out to us in the #mining channel on Discord Arweave Dev Talk server (https://discord.gg/DjAFMJc) or email us on team@arweave.org.

Assets 4

@ldmberman ldmberman released this Nov 28, 2019

**Transaction priority queue**

The feature seeks to optimize transaction propagation. The goals are to maximize the utility for miners and stay within the resource limits (network bandwidth, RAM).

The task can be seen as a Knapsack auction, where miners need to collect transactions with the biggest total reward, subject to the given total size constraint. The choice of the allowed size, in turn, determines the latency.

We use a greedy approximation algorithm, which prioritizes transactions with a higher per-byte reward.

**Improvements**

 - Reduced network traffic. The number of outgoing connections for propagating transactions is now capped globally, not on a per-transaction basis.

**Upgrade instructions**

Shut down the miner. To make sure the shutdown was successful, run `ps aux | grep beam | grep -v grep`. The output should be empty.
Run `git fetch --all --tags && git checkout N.1.9.1.0 && git reset --hard N.1.9.1.0`.
Start arweave-server again with the same arguments as before.

**Support**

For more details on how to run the node, please see our mining guide - https://docs.arweave.org/info/mining/mining-guide. If you have any issues upgrading or would like to know more about the release, feel free to reach out to us in the #mining channel on Discord Arweave Dev Talk server (https://discord.gg/DjAFMJc) or email us on team@arweave.org.
Assets 2

@arweave-kyle arweave-kyle released this Oct 30, 2019 · 65 commits to master since this release

Release N1.9.0.0 is out!

The release comes with a hard fork which activates at height 315700, approximately 2019-11-04 at 09:00 CET.

New features in the release:

  • Improved RandomX hashing performance. The performance boost is up to 300%, depending on the setup.
  • Transaction database upgrade. The transaction index backend has been upgraded to use SQLite, giving a huge performance boost to ArQL/GraphQL query speeds, more efficient indexing, and more flexibility to support future development. The feature is already live on arweave.net.
  • The disk space limit feature has been updated to stop writing new files on disk when the disk partition with the Arweave data folder is 98% full, (this default behavior can be tweaked via the disk_space command line argument).

Other improvements and bugfixes:

  • The general memory footprint for nodes has been reduced.
  • Content-Type filtering has been tightened up, fixing an issue where header splitting with certain transaction tag values was possible.
  • Pooling mechanics adjustments including updated benchmarks.
  • Difficulty adjustment per retarget period will now be capped, in order to further stabilize block times.

Upgrade instructions

This is a big update, so comes with a slightly more involved upgrade process.

Shut down arweave-server. To make sure the shutdown was successful, run ps aux | grep beam | grep -v grep. The output should be empty.

Run git fetch --all --tags && git checkout N.1.9.0.0 && git reset --hard N.1.9.0.0

Now, we can update the dependencies and prepare your machine.

1. Erlang 21

Support for legacy Erlang 20 has been removed, Erlang 21 is now the minimum supported Erlang target. To check your current version, run erl -v, if you're running less than 21, run the following steps.

To upgrade Erlang on Ubuntu 18.04:

wget http://www.erlang.org/download/otp_src_21.3.tar.gz; tar -zvxf otp_src_21.3.tar.gz; cd otp_src_21.3; export ERL_TOP=`pwd`; ./configure --without-wx && make -j4 && sudo make install

2. SQLite

To install the SQLite dependency on Ubuntu 18.04, run the following:

sudo apt-get install libsqlite3-dev

3. Restart the node

Start arweave-server again with the same arguments as before, on the first run a database migration process will start, this process doesn't need to be supervised and can take an hour or so, depending on the specs of your machine. A ar_sqlite3: populated_db log message will be emitted when the upgrade is complete.

Your node will function as normal during this migration process with the exception of the /arql endpoint, which will return a 503 status code.

The install.sh script has been and the mining guide notes have been updated to reflect all of these changes.

Support

For more details on how to run the node, please see our mining guide - https://docs.arweave.org/info/mining/mining-guide. If you have any issues upgrading or would like to know more about the release, feel free to reach out to us in the #mining channel on Discord Arweave Dev Talk server (https://discord.gg/DjAFMJc) or email us on team@arweave.org.

Assets 2

@ldmberman ldmberman released this Oct 17, 2019 · 90 commits to master since this release

The release includes a feature and four improvements and bugfixes.

  • Support for path manifests is introduced

    Path manifests are a simple and optional extension for Arweave gateways; they allow users to upload a small metadata transaction (the path manifest), which maps user-definable subpaths with other Arweave transaction IDs. This allows users to create logical groupings of content, for example, a directory of related files, or the files and assets that make up a web application.

    For more details, check the wiki page about path manifests. It describes the schema and provides an example.

  • The benchmark was made more precise

    Now it collects the actual number of hashes the mining processes generate instead of estimating them from the number of mined blocks. As a result,
    the benchmark fluctuates less and reports slightly higher numbers.

    Link.

  • The propagation of blocks and transactions was improved slightly

    Link.

  • A bug was fixed where nodes would sometimes fail to join the network

    Link.

  • An unstable memory monitoring process was removed that sometimes caused the nodes to stop mining

    Link.

Upgrade instructions:

  1. Shut down arweave-server. To make sure the shutdown was successful, run ps aux | grep beam | grep -v grep. The output should be empty.
  2. Run git fetch --all --tags && git checkout N.1.8.3.0 && git reset --hard N.1.8.3.0
  3. Start arweave-server again with the same arguments as before.

For more details on how to run the node, please see our mining guide. If you have any issues upgrading or would like to know more about the release, feel free to reach out to us in the #mining channel on Discord Arweave Dev Talk server or email us on team@arweave.org.

Assets 2

@ldmberman ldmberman released this Sep 24, 2019 · 109 commits to master since this release

The release contains three improvements.

  1. RandomX tuning

Three RandomX parameters (used to be always off) can be now enabled or disabled via the command line:

  • JIT

    Switched on by default on Linux. Can be disabled via disable randomx_jit.

    The performance boost can be ~ 5x.

  • Hardware AES

    Switched on by default. Can be disabled via disable randomx_hardware_aes.

    The performance improvement is about 30%.

  • Large pages

    Switched off by default. The OS needs to be configured before it can be used.

    On Ubuntu, to see the current values, execute cat /proc/meminfo | grep HugePages.
    To set a value, sudo sysctl -w vm.nr_hugepages=2000. 2000 * 2 MB = 4GB. Consider
    increasing the value if you have more memory. 400 multiplied by the number of logical cores
    you are mining on might work best in case you have so much RAM. Check the RandomX
    repository for more details - https://github.com/tevador/RandomX.

    Should give the performance improvement around 2x.

  1. Fewer forks

The issue was addressed when nodes would reject a height + 1 block from an alternative
chain to incur additional cost for the potential minority to store content, which is
considered illicit by the majority. Nodes now make an explicit content scan and only
wait for yet another block if the scan fails.

  1. Pessimistic pricing

The /price endpoint was made more pessimistic, to ensure almost every accepted transaction
is not invalidated before being included into a block when the network difficulty decreases.

Upgrade instructions:

  1. Shut down arweave-server. To make sure the shutdown was successful, run ps aux | grep beam | grep -v grep. The output should be empty.
  2. Run git fetch --all --tags && git checkout N.1.8.2.0 && git reset --hard N.1.8.2.0
  3. Start arweave-server again with the same arguments as before.

For more details on how to run the node, please see our mining guide - https://docs.arweave.org/info/mining/mining-guide. If you have any issues upgrading or would like to know more about the release, feel free to reach out to us in the #mining channel on Discord Arweave Dev Talk server or email us on team@arweave.org.

Assets 2

@ldmberman ldmberman released this Sep 17, 2019 · 122 commits to master since this release

**New features in the release:**

1. A new interface to query TX metadata using GraphQL.

The main issues addressed by the new interface are:

- Use an existing and well-defined query language to let apps fetch data from the weave
- Reduce the number of roundtrips required to get an exploitable dataset to use in an app on the permaweb
- Allow querying through user-defined relationships between transactions

GraphQL requests are processed by the POST /arql endpoint. The ArQL language
is still supported - the requests are dispatched to the corresponding backend
according to the received query.

The new queries are resolved using the same backing database as the ArQL queries,
meaning that the feature only presents negligible performance implications compared
to the previous implementation.

GraphQL queries are validated against the schema defined in priv/schema.graphql.

2. Webhooks.

A node can be configured to post blocks and transactions to the specified HTTP endpoints.
Failed requests are retried.

**Bugfixes and improvements:**

1. Fixes a race condition during block verification that causes valid blocks
   to occasionally be rejected (~13% of the time), leading to a lower chance of
   forks occurring.
2. The efficiency of block application is improved.

**Upgrade instructions:**

1. Shut down arweave-server. To make sure the shutdown was successful, run `ps aux | grep beam | grep -v grep`. The output should be empty.
2. Run `git fetch --all --tags && git checkout N.1.8.1.0 && git reset --hard N.1.8.1.0`
3. Start arweave-server again with the same arguments as before.

For more details on how to run the node, please see our mining guide - https://docs.arweave.org/info/mining/mining-guide. If you have any issues upgrading or would like to know more about the release, feel free to reach out to us in the #mining channel on Discord Arweave Dev Talk server or email us on team@arweave.org.
Assets 2
Aug 21, 2019
The release comes with a hard fork which activates on 2019-08-29 appr…
…oximately at 14:00.

**New features:**

1. Multiple transactions per wallet per block.

The client-side support (JS SDK, arweave-deploy, browser extensions) is coming soon. In case you want to prepare transactions yourself, query the new GET /tx_anchor endpoint and put the result into the last_tx field of your transaction. You can use the same "anchor" for multiple transactions. Until the fork activates, there is still a limitation of one transaction per wallet per block. For backward-compatibility reasons, we are keeping support for the old way of building transactions using the last transaction of the wallet for last_tx. So the current apps should continue functioning. Note though that only one transaction built the old way per wallet per block is included into the block, even after the fork.

2. Transaction data size limit has increased to 10 MB.

3. Linear difficulty.

Effective from the first block after the fork, the network difficulty is a number a hash must be greater than. `2 ** 256 / (2 ** 256 - Diff)` is an average number of hashes one needs to try to mine a block. The new format allows us to do smooth increments and decrements of the difficulty, proportional to the ratio of the actual time to the target time. As a result, block time becomes more stable.

4. There are updates to the pooling mechanics. We provide the elaborated details on transaction pricing in the yellow paper, which is coming soon.

**Bugfixes and improvements:**

1. The performance of transaction verification was improved to account for the increase in the expected number and data volume of transactions.
2. The GET /wallet/[Addr]/txs(/[EarliestTX) endpoint is now based on the ArQL index and works much faster for the wallets with many transactions.
3. The POST /arql endpoint returns results sorted. arweave.net nodes have been already running this for a while.
4. Various reliability improvements were made to ensure smooth node operation.

**Upgrade instructions:**

1. Shut down arweave-server. To make sure the shutdown was successful, run `ps aux | grep beam | grep -v grep`. The output should be empty.
2. Run `git fetch --all --tags && git checkout N.1.8.0.0 && git reset --hard N.1.8.0.0`
3. Start arweave-server again with the same arguments as before.

For more details on how to run the node, please see our mining guide - https://docs.arweave.org/info/mining/mining-guide. If you have any issues upgrading or would like to know more about the release, feel free to reach out to us in the #mining channel on Discord https://discord.gg/bC5W2T or email us on team@arewave.org.

@ldmberman ldmberman released this Aug 21, 2019 · 135 commits to master since this release

The release comes with a hard fork which activates on 2019-08-29 approximately at 14:00 CET.

**New features in the release:**

1. Multiple transactions per wallet per block.

After the fork, multiple transactions made from the same wallet can be posted to a node and included into one or more subsequent blocks.

The client-side support (JS SDK, arweave-deploy, browser extensions) is coming soon. In case you want to prepare transactions yourself, query the new GET /tx_anchor endpoint and put the result into the last_tx field of your transaction. You can use the same "anchor" for multiple transactions. Until the fork activates, there is still a limitation of one transaction per wallet per block. For backward-compatibility reasons, we are keeping support for the old way of building transactions using the last transaction of the wallet for last_tx. Subsequently, the current apps continue functioning. Note though that only one transaction built the old way per wallet per block is included into the block, even after the fork.

2. Transaction data size limit has increased to 10 MB.

3. Linear difficulty.

Effective from the first block after the fork, the network difficulty is a number the hash must be greater than. `2 ** 256 / (2 ** 256 - Diff)` is an average number of hashes one needs to try to mine a block. The new format allows us to do smooth increments and decrements of the difficulty, proportional to the ratio of the actual time to the target time. As a result, block time becomes more stable.

**Bugfixes and improvements:**

1. The performance of transaction verification was improved to account for the increase in the expected number and data volume of transactions.
2. The GET /wallet/[Addr]/txs(/[EarliestTX) endpoint is now based on the ArQL index and works much faster for the wallets with many transactions.
3. The POST /arql endpoint returns results sorted. arweave.net nodes have been already running this for a while.
4. There are updates to the pooling mechanics.
5. Various reliability improvements were made to ensure smooth node operation.

**Upgrade instructions:**

1. Shut down arweave-server. To make sure the shutdown was successful, run `ps aux | grep beam | grep -v grep`. The output should be empty.
2. Run `git fetch --all --tags && git checkout N.1.8.0.0 && git reset --hard N.1.8.0.0`
3. Start arweave-server again with the same arguments as before.

For more details on how to run the node, please see our mining guide - https://docs.arweave.org/info/mining/mining-guide. If you have any issues upgrading or would like to know more about the release, feel free to reach out to us in the #mining channel on Discord https://discord.gg/bC5W2T or email us on team@arweave.org.
Assets 2
You can’t perform that action at this time.