---------------------------------------------------------
SUMMARY AND OUTLOOK - detailed changelog is below:
The main goal of this build is to improve the very poor latency of the
web interface.
This is done by redesigning the way fetched Identity XML files are
processed.
Previously, each file would be processed in a thread of its own, even
though the actual processing requires a single lock which can only be
taken by one thread at once. This did cause processing to be effectively
single-threaded, while hundreds or even thousands of threads would be
piling up when competing for the single lock.
With this build, processing is redesigned to queue fetched XML files to
disk and process them one-by-one on a single thread.
As a result, the web interface thread now does not have to compete
against hundreds of threads who want the same lock.
This will result in much lower latency of the web interface, especially
if your WOT is not permanently running and thus at startup has to
process many identity files in bulk.
There are some more benefits which are listed in the detailed changelog
below.
It is still possible for the web interface to stall for minutes due to
the fact that the computations to import a trust list are still
implemented with a very poor algorithm.
The goal of the next build is to improve this algorithm significantly.
The next build is feature-complete already and will be available as
release candidate soon.
Also, please notice that usage of client applications which have not
been adapted to use the new WOT event-notifications API yet can slow
down the web interface significantly. This applies to especially Sone
which downloads almost the whole WOT database every 60 seconds [1].
If you are a client application developer and want to make usage of
event-notifications, please contact me on IRC (nick "xor-freenet"), I am
willing to help by explaining how to use the reference implementation
FCP client.
Some builds later on, the locking on the web interface will be removed
completely to allow it to always render instantaneously. This will take
a lot of time to implement due to issues with the database and thus has
been postponed to after the more easy algorithm improvements.
In addition to the performance fixes, many new translations of the UI
are included: 9 new languages, and 3 updated ones.
Very very many thanks to the contributors on Transifex, and to operhiem1
for mentoring them!
CHANGELOG - prefixed with the bugtracker issue number:
- 0006244: [Performance] Queue fetched trust lists instead of processing
them immediately (xor / nextgens)
This is what has been described in the introduction.
Beyond the latency improvement of the web interface, this also:
- Helps plugins such as FlogHelper which have not been adapted to
event-notifications yet. They should lose connection to WOT less
frequently, which fixes issues such as FlogHelper saying that WOT
is not loaded even though it is.
- Helps users of machines with little memory such as Raspberry Pi.
Less threads = less memory usage.
- Speeds up shutdown
- Reduces total CPU/IO usage by identity file deduplication: If an
identity file is queued for processing with edition N, and the node
fetches edition N+1 of that identity before edition N has been
processed, then the old edition N will be dropped. Only the newest
will be processed. It is possible that deduplication does not yield
a good ratio yet. To fix this, more measurements will have to be
done in the future to tweak the time window for which we wait before
processing a queued file. Generally, on slower machines there will
be more deduplication, which is a nice side effect as deduplication
reduces load and slow machines can use that.
To compute the deduplication ratio, go to the "Statistics" page
and divide "Deduplicated files" by "Total ever enqueued (= fetched)
files". Feel welcome to share this number with the team.
- Helps make WOT look less buggy to new users: They were the ones
where the most new XML files would be fetched at once, and thus
did suffer the most severe web interface lockup.
- Persists fetch-progress across restart: Previously, the progress of
the stalling threads would be lost at shutdown. The new queue code
will keep the queued files across restarts.
- Allows creation of "bootstrap" bundles of a dump of the XML files of
all Identities to help new users with slow connections.
See below at issue 006597. (Please do read this, there is a security
pitfall with this feature!)
- Helps developers to deterministically test changes to WOT,
see below at issue 0006597.
You can view statistics about the new identity file queue and identity
file processor on WOT's "Statistics" page.
In particular, the two "Failed files" counters are of interest.
Consider looking at your log files if they are non-zero.
- 0006561: [Usability] L10n: Add new translations from Transifex, update
existing ones (operhiem1, and many Transifex contributors)
0006560: [Security] L10n: Review German translation on Transifex (xor)
0006495: [Usability] L10n: Update Transifex, new l10n strings have
been added (operhiem1)
New translations:
Brazilian Portugese
Finnish
French
Greek
Hungarian
Norwegian
Polish
Simplified Chinese
Spanish
Updated translations:
Dutch
German (also fully reviewed and improved compared to English, now
can be considered as reference translation)
Italian
- 0006499: [Security] Web interface: Allow sorting identities by edition
count (xor)
This allows you to check for identities which insert an excessive
amount of trust lists, which can be considered as a denial of service
attack.
If an identity does this, please try to contact them to find out
whether it is a bug in their WOT; and distrust them if do not
cooperate to fix this behavior.
Also, report this to the Freenet team please so we can check for bugs.
- 0006554: [Code quality] Web interface / Statistics: Add "Number of
unfetched identities" (xor)
For first time users or people whose WOT has been offline for a long
time, this number can be used to find out how far WOT is with
downloading all identities.
- 0006230: [Security] Make sure to always display the logged-in identity
in the web interface (xor)
This is to prevent you from accidentally revealing which of your
identities are owned by the same person by setting a trust value
or trust comment with the wrong identity.
For now, it will show the full ID of the identity, which is very long
and thus ugly.
Fixing this is a bit of work and thus it will have to wait until
after performance is fixed.
See https://bugs.freenetproject.org/view.php?id=6072
- 0006493: [Usability] Web interface: Change page title from
"Web of Trust" to "Community"; allow translating (xor)
This is to ensure that the page title matches the name of the
"Community" menu through which WOT is accessed.
- 0006491: [Performance] Update seed identity edition numbers
immediately before releasing (xor / ArneBab)
This will speed up downloading all identities for new users.
CHANGELOG about stuff only interesting for developers:
- 0006597: [Code quality] IdentityFileDiskQueue: Support deterministic
repeat by dequeuing sorted by filename (xor)
The new disk queue for identity files, which was described above at
issue 0006244 already, has a feature for repeating test runs:
- If DEBUG logging is enabled, it will archive files after processing
- to the directory "IdentityFileQueue/Finished".
The filenames will be prefixed with an index number to indicate the
order in which they were processed.
- If DEBUG logging is enabled, it will process files in the "Queued"
directory sorted by filename instead of in random order.
This means you can copy files from the "Finished" to the "Queued"
directory to repeat a test run in a deterministic way - the index
number prefix together with the sorted import will guarantee that
the files will be imported the same order as in the original run.
Please shutdown WOT before doing this.
The deterministic test runs are a useful feature for:
- Reproducing bugs in a debugger.
- Benchmarks: The statistics page shows the total time the identity
file processor spent importing files.
- Creating "bootstrap" dumps of Identity files of all Identities so
new users can add them to their queue instead of waiting for them
to be fetched. Please be aware that the cryptographic signatures of
the files are NOT verified and thus you put a lot of trust in the
person who provides the bundle - the files can be modified in any
way.
- 0006531: [Code quality] Unit tests which can fail on slow machines
should only be executed if "test.unreliable=true" (xor)
This applies to the tests in package "plugins.WebOfTrust.util.jobs".
They test a timer class, and thus are timing-sensitive and fail
randomly on slow or heavily loaded machines.
- 0006523: [Code quality] Ant build improvements (xor)
The "-projecthelp" parameter now lists all build targets. It was
missing "junit".
It now also lists the various "-D" options for junit.
The "-D" options for junit now all start with "test.". Namely,
"-Dbenchmark=true" is now "-Dtest.benchmark".
- 0006522: [Code quality] Merge PR 28: Ant build: allow for
-Dtest.class=... unit test selection (bertm / xor)
This allows running only a single unit test class from Ant. It's
useful if you are working on a new test class for example.
Thanks to bertm for implementing this!
- 0006368: [Bugs] Identicon leaks an InputStream (xor)
This memory leak would only apply when using a WOT-based application
which showed Identicons via WOT. This probably applied to Sone.
Thanks to:
- nextgens for motivating me to implement the performance optimization
which is the main feature of this build.
- operhiem1 for administrating Transifex which has proven to be a huge
source of new Translations; and for committing the new l10n files.
- ArneBab, stanley and probably some more people I forgot about for
testing
- bertm for the Ant improvements
- you, the users, for having patience while performance optimizations
are implemented one after another.
Links:
[1] https://github.com/Bombe/Sone/pull/11
