Google Summer of Code 2014
Table of Contents
- Improve the web interface generally
- There is much to do to make it more user friendly. We will hopefully have a set of mock-up designs soon, but there are also many small things linked from this bug report; many are not, please have a look around, also check the mailing list archives. A recent but not very detailed, professional mockup focusing on the homepage is here: https://emu.freenetproject.org/freenet-ui-mockup-pupok-1.pdf. An older detailed suggestion might also be an inspiration. There was quite a bit of work on this in 2012 though the project didn't succeed; it's worth looking at the devl archives etc.
- Get the web-pushing mode working well
- A previous Summer of Code student built the web-pushing ajax/comet/GWT system, which does things like showing progress on pages with lots of images to speed things up when browser connections are limited, and live updating of the downloads page etc. This is rather buggy especially on slow browsers, and tends to lock up, so fixing this, improving it and then extending it to other areas (e.g. Freetalk) would be interesting. Obviously rewriting the whole web interface with a different toolkit would need a different Comet solution, but we'd like to have something like this.
For some projects in this section, the best solution would simply use an external library to do most/much of the work, and simply integrate it into the current freenet code. So you should do some research into existing solutions first. This principle applies to all projects, but the ones where suitable libraries are most likely to actually exist, are tagged prefer-external.
SSKs currently use our own implementation of DSA. This is old, and the pubkeys are huge. We can switch to ECC, using the Java Cryptography Architecture.
- Make SSKs use ECC for signatures, via JCA (see below).
- Get NSS working semi-automatically. For ECC this is a big performance gain.
- Take advantage of the smaller size of ECC-based SSKs' pubkeys: optimise SSK requests and inserts (fewer packets etc).
- Take advantage of the smaller size of ECC-based SSKs' pubkeys: optimise datastore usage (single store for SSKs and pubkeys), and allow keys of ~ 800 bytes (fast, for e.g. IRC over Freenet) and ~ 2KB (slower, for e.g. forums). Automatic, safe migration, even if there are disk space issues, is one of the bigger problems here.
- Implement 32KB SSKs as well, if we're looking at the datastore.
- Possibly implement Programmable Subspace Keys. The basic principle is to allow to specify a set of rules (in a simple language) that determine whether or not a block is valid for a given key. The simplest application of this is a way to allow multiple posters to use a single SSK (greatly increasing the efficiency of a spam-proof freenet-based chat system), while still requiring them to sign their own posts, allowing the "moderator" to change the key to get rid of a spammer.
This project is about changing how we do cryptography in freenet. At the moment, we use our own-implementation of standards... This project is about switching to JCA and BouncyCastle.
JCA has cryptographic strength restrictions, and even for US users resolving this is nontrivial (the java auto-updater wipes out the unlimited strength policy files every time you update!), so we will need to implement some internal API that can wrap either our built-in crypto, Bouncycastle's lightweight API, or an accelerated JCA implementation. This is probably a good idea anyway, to make the code clearer and limit the possibilities for accidental abuse of cryptographic primitives. In particular, crypto setup in connection setup should be factored out into a separate class, and we should consider using third party libraries for parsing the connection setup packets (and possibly for parsing messages too, the same library could do both tasks).
Another problem is that native implementations are often slower, because of the overhead of calling C code through JNI. This may not be true on all hardware (e.g. ATOM, ARM), so we need to benchmark it on startup (this is partly implemented already). For security reasons it is preferable to use JCA, or at least Bouncycastle, where possible, if there is no performance issue.
The major goals of this project are:
- Review and question the usage of cryptography thorough the whole codebase
- Build an internal, high level crypto API, which wraps either Bouncycastle lightweight API or JCA.
- Use this instead of our own code, reducing the amount of crypto code we need to keep in fred.
- Benchmark or profile hot codepaths and propose enhancements
- Enable users to use crypto-accelerators where possible (for x86 systems this probably means asymmetric i.e. ECC; symmetric crypto appears to be slower even with AESNI because of the JNI overhead).
- Write unit tests to ensure some form of backward compatibility with the existing code (there are some already, more would be good)
The current updater makes some very specific assumptions as to the locations of the jars, and this method doesn't generalize to non-jar files. It also assumes wrapper.conf is in the current directory. This is at odds with attempts to package Freenet according to FHS. (There is already a debian package but currently this disables auto-update. Even though this is likely to remain the case due to Debian policy, we can use this package as an example to test out the enhanced auto-updater on.)
The auto-updater should be able to:
- update jars, freenet.jar and freenet-ext.jar, and the various 3rd-party dependency libraries. (Currently said libraries are packaged inside freenet-ext.jar, but the newer version separates them out. The current auto-updater can't handle this, so we can't yet deploy the newer freenet-ext.)
- update any native binaries/libraries (e.g. the wrapper files)
- the above probably means having some sort of manifest that lists which files are actually installed. If this is the case, all of our installers need to be change to include such a manifest too.
Freenet uses a custom-written framework for running callbacks and scheduled tasks, which is non-standard and lacks a lot of advanced functionality. Converting everything to use java.util.concurrent and/or com.google.common.util.concurrent would help this a lot. Ideally, the framework would be separated from the rest of freenet, possibly even to the point of being an external library, and provide a clean API for freenet and its plugins to use. This would also allow plugins to do much more advanced tasks (e.g. most of the things in the "plugins and applications" section further below) without having to implement their own framework.
The events framework should be able to, and provide a good API for plugins to:
- run tasks in the background, or according to a particular schedule
- define task+thread matching policies e.g. all database tasks must run in the "database" thread
- cancel running tasks, if the task defines that it's cancellable
- handle dependencies between tasks, so that e.g. if A depends on B and we cancel/complete B, A is automatically cancelled/completed.
- attach callbacks to events such as task completion/cancellation (e.g. using com.google.common.util.concurrent.ListenableFuture)
- group tasks into related categories so one group doesn't affect the other. (e.g. tasks for the group "Library/b-tree-write/index-" won't starve the resources of group "WoT/background-trust-calculation") (this is possibly logically equivalent to the "matching policies" point above but I haven't thought too much about it)
Currently, config management code is very untidy. Code for read/writing the config in embedded inside the classes they control, which is spread out over many files not related to config management. This creates lots of implicit dependencies, which is not directly apparent in the syntax/structure of the code nor the semantics of the language. (In other words, spaghetti code.)
Config code should be separated from the actual application. It would also be nice if there was an API to expose this to plugins.
The config system should be able to:
- support everything the current config system supports
- use "includes" to separate different sections into different files - e.g. keep vital system-specific settings in a separate read-only file
- read and write values, preserving comments in the file.
- With regards to mutability, there are three types of values:
- mutable (takes effect after a restart)
- mutable (effects the running daemon immediately)
- Note that the actual code for doing the setting should be implemented outside of the config system.
- With regards to mutability, there are three types of values:
operhiem1 addressed the infrastructure parts of this project in Google Summer of Code 2013 by extending the Infocalypse plugin. He simplified sharing code anonymously to:
- hg clone freenet:ID/reponame
- cd repo
- hg push freenet:MyID/reponame
- hg fn-pull-request
- hg fn-check-notifications`
Thus as followup, this project would be turning the existing infrastructure into a polished workflow which makes hacking freenet in freenet enjoyable.
- Add discovery of repositories and connections between repositories by browsing the Web of Trust: “find forks of my repo in freenet”. Offer the option `hg paths --add-forks` (or similar), which shows all public forks in freenet.
- Create freesites with information about the repositories. See the hg site extension for an example.
- Enhance the DVCS web interface plugin to provide nice repository browsing capabilities.
- Add automatic tests for selected collaboration workflows.
- Add communication features, discussion of pull-requests and optionally other parts needed for a full fledged community hacking platform.
- Optionally integrate Infocalypse with the keepalive plugin to improve the longevity of repositories.
The current website infrastructure is a mess of PHP and shell scripts that's difficult to update and localize. Something that's more flexible and can be deployed locally for testing would be very useful.
- Improve Sone
- Sone is a social-networking style system over Freenet. One way to improve it is to make it automatically insert bundles of posts periodically, to avoid having to download all the posts every time. Other problems relate to the question of scalability: When you can't follow everyone, what does that mean for the user interface? How do you do keyword searches? Etc. There have been ideas on this. Talk to Bombe...
- A good filesharing/file search system
- This should tie in with the Web of Trust, allowing users to publish indexes and search those of their anonymous friends, rate others' indexes, merge them into their own, set up long-term file searches, preload indexes for faster searches, and so on. It might also integrate with Freetalk to help with discussions on labelling or rating. The problems of spam/deliberately corrupt content are very similar on Freenet to on traditional p2p, although the solutions may be different, especially as it isn't possible to trace spammers; trusted community maintained indexes have developed as a working means of solving these problems on web-based filesharing. Note that we already have a scalable forkable on-freenet btree search system to use as a backend, but it is not yet used for anything, and it is not distributed or WoT-compatible.
- Another interesting area for filesharing is a distributed, WoT-based way to download data by conventional hashes rather than CHKs, which could tie in with other networks; this is also related to the wierd stuff (backups) at the bottom.
- Secure reinsert-on-demand filesharing, to improve the volume of content that is available. This is a lot harder than it sounds, but in any case we need searching first. (Reinserts using the same keys are seriously insecure, although getting the downloaders to do random inserts may help significantly)
- Keepalive plugin improvements
- Some very small changes at the node layer will allow probing for whether a key is available from a random node. Users could then maintain a list of content that they care about; the plugin would download it all as a binary blob, regularly probe for reachability, and reinsert it when needed (possibly single random not fetchable blocks). For this to work really well we might need selective reinsert support in the client layer, but that's not necessary for a basic implementation. What is important is supporting both files and sites, and letting users publish their lists and subscribe to others. This could even evolve into a full blown distributed backup system, with configurable encryption for files and the ability to recognise which files are popular and avoid inserting them (might need fixing GetCHKOnly).
- A microblogging and/or real-time chat system
- Both of these things would actually be implemented in a fairly similar way. Evan has done a fair amount of work on how to efficiently implement microblogging over Freenet. Sone does something like this but is fairly greedy with network resources. Flip does IRC.
- FCP libraries
- Good FCP libraries in more languages.
- Easy-to-use tools
- We need simple tools for inserting freesites (freenet-hosted web sites) and files, and maintaining mirrors of web content: We already have a blogging tool, but it needs more work, and tools to make it easy to insert existing content etc would also be useful. This should support uploading files of any size, should avoid re-uploading larger files on every update, but should be configurable to do so on a schedule, should work from within the freenet web interface as a plugin, and may support WebDAV uploads direct from authoring software. The ability to mirror stuff from the web would also be useful.
- Scalable fork-and-merge distributed revision control over Freenet
- This would integrate the new scalable on-Freenet b-trees from the new Library format by infinity0, in order to scale up to at least Wikipedia scales (to implement a wiki over Freenet using a fork-and-merge model). It would tie in closely with the Web of Trust (the trust network backing Freetalk), integrating with its identities and announcing forks, and allowing users to easily see changes in other forks and integrate them. The most obvious use for this is a wiki-over-freenet (note that because of spam and denial of service attacks, instant anonymous editing of a wiki on freenet is not possible), it might also be useful for distributing spidering Freenet, for source code (e.g. if we want to deploy a new build only after a certain number of people we trust have signed it, and then build it from source), or for anything that needs a forkable database over Freenet. You might also need to optimise the btrees' data persistence by e.g. including the top level metadata for each chunk in the next layer up. Scalable fork and merge databases are closely related to this. Infocalypse already provides a working plugin for efficient sharing of Mercurial repositories, but it needs integration with the Web of Trust, and added efficiency through btrees would be useful.
- Better freesite searching
- Lots of work has been done on this, but more could be done: Using the new library format, rewriting the indexes on the fly after gathering a few hours' data rather than writing it from the database over a week, support for long-term searches, web of trust integration, better support for stop-words (maybe aggregating them with common before/after words), tokenisation for tricky languages (Chinese, Japanese), distributing spidering across multiple users (as scaling is getting to be a serious problem now), etc.
- Wiki over Freenet
- A wiki over Freenet would be really awesome. In fact it could be a killer app. But it is not easy to implement, as there are several challenges. You can learn more there. There have been many attempts; some are hard to use and based on DSCMs, some are easier to use and not scalable.
- WebOfTrust optimization
- Currently the Web of Trust plugin is quite slow. Optimizations on the database level and on synchronization strategies should offer the chance it much faster. A first step would be profiling to find the real performance bottlenecks. Care will be needed to ensure using scalable algorithms. There is an alternative Web of Trust implementation called LCWoT, but it is incomplete. Note the next item.
- Use PSKs (see the section on SSKs and ECC)
- We can use these to implement a much more scalable, but still spam-proof, chat forum system: This may or may not use WoT. Announcement should be per-context, that is, users should announce to a single forum, but should be visible to all users that watch the forum, rather than posting a message to a few users. Ways to do this, and to limit the problem of users filling in the same CAPTCHAs, have been posted on devl and in the bug tracker. Actual boards should be via PSKs: In the first instance, a moderator may expel a spammer by issuing a new key and signing the non-spamming users' pubkeys; in a more decentralised system, each user is a moderator, and we can detect when they abuse their powers by blocking non-spammers (here trivial spam isn't the problem, sustained DoS attacks are).
- More content filters
- More simulations
- There is lots that can be simulated: Does swapping scale? Does the proposed fix for Pitch Black really work? Do darknet and opennet get on together? In particular, there are a range of attacks on requests that have been proposed, it would be great to quantify them. Load management is another huge area where simulations (and expertise!) would be helpful.
- Tunnels and so on
- Somebody with some expertise in tunnels would be very useful. Freenet needs an optional mixnet layer for better security, especially on inserts (where it wouldn't cost us much performance because we go lots of hops anyway). We have some ideas (something based on Pisces perhaps?)... This is a more research-oriented project but we would like to have some sort of basic tunneling option available by the end of it.
- Much more transport layer stuff
- The current transport layer has been improved significantly but leaves much room for improvement. Ideally we'd like to detect available bandwidth automatically. We have nothing remotely like Path MTU detection; we should automatically adapt to find the right packet size for the connection, both finding what will work at all, and what gives the best bandwidth/loss tradeoff. We tend to get transfer failures on slow connections (not low bandwidth limit, low bandwidth available on that specific connection). We probably should use something like cumulative acks, currently all packets are acked once, it should be possible to ack the same packet twice with 0 bandwidth cost in many cases using ranges. We may want to divide up blocks differently depending on how fast the connection is. We may want to make tradeoffs between fairness to all peers (the current policy) and allowing individual peers more bandwidth for a short period (e.g. because they have requested a bunch of fproxy pages), or have "idle priority" traffic which is only sent when *no* peer wants to send anything (e.g. bloom filter sharing), which may also impact on packet size. And so on. Generally, the transport layer needs to be more robust, especially on slow connections, and it needs to feed information into the load management layer more quickly so that we only accept requests that we can complete in a reasonable time, given the current state of the connection. There are various bugs about this on the bug tracker. Running as well as possible on fairly slow connections is particularly useful in some of the places where Freenet may be needed most.
- Chetan's work in 2012 means we will soon have a general layer for packet-type transport plugins, but we won't have any actually working yet. We would like support for stream-based transport plugins, and a bunch of working transport plugins, for both making Freenet work behind nasty firewalls, and for steganography. Some, such as something that looks like Skype traffic, will impose particular transport-layer problems due to e.g. packet sizes.
- Different bandwidth limits at different times of the day/week would help many users. We also need support for monthly transfer limits (separate from the existing peak per second limits), and auto-detection of the connection's capacity, possibly with options for very latency sensitive behavior for e.g. gamers (like some Bittorrent clients do). All this must be really easy to use.
- Better darknet
- Make it easier to invite people to Freenet, provide software bundles, provide a simple password-and-IP-address connect protocol, connect to friend-of-a-friends, make it easy to connect to mutual friends, etc. Lots of ideas about this on the bug tracker. Some of it has been implemented in a branch (foaf-connections).
- More F2F functionality
Tahrir is a sister project to Freenet, incorporating some of the same ideas, although focussed more on "twitter"-like functionality (or, more generally, microblogging). Much of Tahrir's infrastructure is complete, including a powerful but easy to use low-level transport layer, together with crypto functionality, however the components need to be brought together into a working prototype. More information at The Tahrir Project.
You can sign up between March 10th and March 21st on the Google Summer of Code page.