Google Summer of Code 2012
Table of Contents
- Improve the web interface generally
- There is much to do to make it more user friendly. We will hopefully have a set of mock-up designs soon, but there are also many small things linked from this bug report; many are not, please have a look around, also check the mailing list archives. A recent but not very detailed, professional mockup focusing on the homepage is here: https://emu.freenetproject.org/freenet-ui-mockup-pupok-1.pdf. An older detailed suggestion might also be an inspiration.
- Get the web-pushing mode working well
- A previous Summer of Code student built the web-pushing ajax/comet/GWT system, which does things like showing progress on pages with lots of images to speed things up when browser connections are limited, and live updating of the downloads page etc. This is rather buggy especially on slow browsers, and tends to lock up, so fixing this, improving it and then extending it to other areas (e.g. Freetalk) would be interesting. *Note:* We have recently had someone volunteer to rewrite our entire web interface using GWT, which would obviate the need for web-pushing mode - please consult the mailing list about this.
For some projects in this section, the best solution would simply use an external library to do most/much of the work, and simply integrate it into the current freenet code. So you should do some research into existing solutions first. This principle applies to all projects, but the ones where suitable libraries are most likely to actually exist, are tagged prefer-external.
Switch from the current code to JCA
This project is about changing how we do cryptography in freenet. At the moment, we use our own-implementation of standards... This project is about switching to JCA and BouncyCastle.
The goal of the project is:
- Review and question the usage of cryptography thorough the whole codebase
- Switch from our own java (or JNI) implementation to a JCA equivalent
- Benchmark or profile hot codepaths and propose enhancements
- Enable users to use crypto-accelerators where possible
- Write unit tests to ensure some form of backward compatibility with the existing code
The current updater makes some very specific assumptions as to the locations of the jars, and this method doesn't generalise to non-jar files. It also assumes wrapper.conf is in the current directory. This is at odds with attempts to package Freenet according to FHS. (There is already a debian package but currently this disables auto-update. Even though this is likely to remain the case due to Debian policy, we can use this package as an example to test out the enhanced auto-updater on.)
The auto-updater should be able to:
- update jars, freenet.jar and freenet-ext.jar, and the various 3rd-party dependency libraries. (Currently said libraries are packaged inside freenet-ext.jar, but the newer version separates them out. The current auto-updater can't handle this, so we can't yet deploy the newer freenet-ext.)
- update any native binaries/libraries (e.g. the wrapper files)
- the above probably means having some sort of manifest that lists which files are actually installed. If this is the case, all of our installers need to be change to include such a manifest too.
freenet uses a custom-written framework for running callbacks and scheduled tasks, which is non-standard and lacks a lot of advanced functionality. Converting everything to use java.util.concurrent and/or com.google.common.util.concurrent would help this a lot. Ideally, the framework would be separated from the rest of freenet, possibly even to the point of being an external library, and provide a clean API for freenet and its plugins to use. This would also allow plugins to do much more advanced tasks (e.g. most of the things in the "plugins and applications" section further below) without having to implement their own framework.
The events framework should be able to, and provide a good API for plugins to:
- run tasks in the background, or according to a particular schedule
- define task+thread matching policies e.g. all database tasks must run in the "database" thread
- cancel running tasks, if the task defines that it's cancellable
- handle dependencies between tasks, so that e.g. if A depends on B and we cancel/complete B, A is automatically cancelled/completed.
- attach callbacks to events such as task completion/cancellation (e.g. using com.google.common.util.concurrent.ListenableFuture)
- group tasks into related categories so one group doesn't affect the other. (e.g. tasks for the group "Library/b-tree-write/index-" won't starve the resources of group "WoT/background-trust-calculation") (this is possibly logically equivalent to the "matching policies" point above but I haven't thought too much about it)
Currently, config management code is very untidy. Code for read/writing the config in embedded inside the classes they control, which is spread out over many files not related to config management. This creates lots of implicit dependencies, which is not directly apparent in the syntax/structure of the code nor the semantics of the language. (In other words, spaghetti code.)
Config code should be separated from the actual application. It would also be nice if there was an API to expose this to plugins.
The config system should be able to:
- support everything the current config system supports
- use "includes" to separate different sections into different files - e.g. keep vital system-specific settings in a separate read-only file
- read and write values, preserving comments in the file.
- With regards to mutability, there are three types of values:
- mutable (takes effect after a restart)
- mutable (effects the running daemon immediately)
- Note that the actual code for doing the setting should be implemented outside of the config system.
- With regards to mutability, there are three types of values:
Integrating DVCS via Infocalypse (Mercurial) with the Web of Trust
Currently code sharing with infocalypse is still a bit inconvenient. Cloning and sharing a repository requires: `hg init; hg fn-pull --uri USK@…; hg up; (hack); hg fn-create --uri USK@/mycode.R1/ ; (share the public key somehow)`
This should be simplified to: `hg clone freenet:ID/reponame; cd repo; (hack); hg push freenet:ID/reponame`
The ID should be the name of a given WoT ID. If that name is not unique, any number of letters from its public key can be added. The URI scheme is an extension of what hg-git offers, so it is doable in finite time. The bigger task would be to add clean integration with the WoT.
Also infocalypse needs some additions to its USK handling, to ease discovery of new revisions.
Simplified URI schemes would be `hg clone freenet:reponame`: If an ID has the repo, get it, if multiple IDs have a repo of that name, ask which one to take.
The overall goal is turning freenet with infocalypse into a decentral, anonymous, community hacking platform by connecting it over the Web of Trust.
- Clone, pull and push via the freenet:key/repo scheme.
- As key allow simple USK keys or WoT IDs.
- If a WoT ID is used, advertise the repo info in the WoT ID.
- Optionally provide additional convenience measures, like automatically creating freesites with information about the repository. See the site extension for an example.
- Optionally add pull requests and anything else needed for a full fledged community hacking platform.
- Optionally: “find forks of my repo in freenet”. Infocalypse would have to register a WoT context, which contains the public key as well as the public key the repo was cloned from (if it was cloned from freenet). Then any plugin could connect the forks again. It could actually offer the option `hg path --add-forks`, which gets all public forks in freenet.
- A good filesharing/file search system
- This should tie in with the Web of Trust, allowing users to publish indexes and search those of their anonymous friends, rate others' indexes, merge them into their own, set up long-term file searches, preload indexes for faster searches, and so on. It might also integrate with Freetalk to help with discussions on labelling or rating. The problems of spam/deliberately corrupt content are very similar on Freenet to on traditional p2p, although the solutions may be different, especially as it isn't possible to trace spammers; trusted community maintained indexes have developed as a working means of solving these problems on web-based filesharing. Note that we already have a scalable forkable on-freenet btree search system to use as a backend, but it is not yet used for anything, and it is not distributed or WoT-compatible.
- Another interesting area for filesharing is a distributed, WoT-based way to download data by conventional hashes rather than CHKs, which could tie in with other networks; this is also related to the wierd stuff (backups) at the bottom.
- Secure reinsert-on-demand filesharing, to improve the volume of content that is available. This is a lot harder than it sounds, but in any case we need searching first. (Reinserts using the same keys are seriously insecure, although getting the downloaders to do random inserts may help significantly)
- Data retention plugin
- Some very small changes at the node layer will allow probing for whether a key is available from a random node. Users could then maintain a list of content that they care about; the plugin would download it all as a binary blob, regularly probe for reachability, and reinsert it when needed (possibly single random not fetchable blocks). For this to work really well we might need selective reinsert support in the client layer, but that's not necessary for a basic implementation. What is important is supporting both files and sites, and letting users publish their lists and subscribe to others. This could even evolve into a full blown distributed backup system, with configurable encryption for files and the ability to recognise which files are popular and avoid inserting them (might need fixing GetCHKOnly).
- A microblogging and/or real-time chat system
- Both of these things would actually be implemented in a fairly similar way. Evan has done a fair amount of work on how to efficiently implement microblogging over Freenet. Sone does something like this but is fairly greedy with network resources. Flip does IRC.
- FCP libraries
- Good FCP libraries in more languages.
- Easy-to-use tools
- We need simple tools for inserting freesites (freenet-hosted web sites) and files. We already have a blogging tool, but it needs more work, and tools to make it easy to insert existing content etc would also be useful. This should support uploading files of any size, should avoid re-uploading larger files on every update, but should be configurable to do so on a schedule, should work from within the freenet web interface as a plugin, and may support WebDAV uploads direct from authoring software. The ability to mirror stuff from the web would also be useful.
- Scalable fork-and-merge distributed revision control over Freenet
- This would integrate the new scalable on-Freenet b-trees from the new Library format by infinity0, in order to scale up to at least Wikipedia scales (to implement a wiki over Freenet using a fork-and-merge model). It would tie in closely with the Web of Trust (the trust network backing Freetalk), integrating with its identities and announcing forks, and allowing users to easily see changes in other forks and integrate them. The most obvious use for this is a wiki-over-freenet (note that because of spam and denial of service attacks, instant anonymous editing of a wiki on freenet is not possible), it might also be useful for distributing spidering Freenet, for source code (e.g. if we want to deploy a new build only after a certain number of people we trust have signed it, and then build it from source), or for anything that needs a forkable database over Freenet. You might also need to optimise the btrees' data persistence by e.g. including the top level metadata for each chunk in the next layer up. Scalable fork and merge databases are closely related to this. Infocalypse already provides a working plugin for efficient sharing of Mercurial repositories, but it needs integration with the Web of Trust, and added efficiency through btrees would be useful.
- Better freesite searching
- Lots of work has been done on this, but more could be done: Using the new library format, rewriting the indexes on the fly after gathering a few hours' data rather than writing it from the database over a week, support for long-term searches, web of trust integration, better support for stop-words (maybe aggregating them with common before/after words), tokenisation for tricky languages (Chinese, Japanese), distributing spidering across multiple users (as scaling is getting to be a serious problem now), etc.
- Wiki over Freenet
- A wiki over Freenet would be really awesome. In fact it could be a killer app. But it is not easy to implement, as there are several challenges. You can learn more there. There have been many attempts; some are hard to use and based on DSCMs, some are easier to use and not scalable.
- WebOfTrust optimization
- Currently the Web of Trust plugin is quite slow. Optimizations on the database level and on synchronization strategies should offer the chance it much faster. A first step would be profiling to find the real performance bottlenecks. Care will be needed to ensure using scalable algorithms.
- More content filters
operhiem1 intends to address this for Google Summer of Code 2012.
Freenet is intended to form a topology which allows efficient local path selection. Unfortunately, currently next to nothing is known about the actual link length distribution. This makes it very difficult to improve as it is only measured indirectly. It is therefore proposed to expose over FCP the probe functionality which is currently undocumented and only available on TMCI, expand probe functionality, and write tools to more thoroughly and quickly present information gathered by the probes.
- Clean up the existing probe code, both to document, improve readability, and correct errors and annoyances:
- Reported peer location overlap: peer locations are reported in an ambiguous overloaded fashion:
- 1 means either location 0 not backed off or an unknown location
- 0 means either location 0 or an unknown location backed off.
- Using a Location object rather than a double which encodes two booleans (backed off or not, known or not) and a double.
- Probe traces can be recieved after a probe completition message is sent. Waiting a while for more traces would be nice.
- Reported peer location overlap: peer locations are reported in an ambiguous overloaded fashion:
- Document probe functionality and make it available over FCP.
- Documentation would make it easier to use and allow its function to be better defined, which would also help in writing and improving it.
- Plot network attributes and make graphs of network topology revealed through probe requests.
- Node churn
- Network size
- Link length distribution
- Graph topology
- Implement improved probe and trace requests as in bugs #3568 and #3550 to provide more detailed information on network structure, health, and behavior.
- Flag to cache only, not store
- would allow transient messages such as IRC (FLIP) from taking storage space.
- No RecentlyFailed on realtime queue
- Much more transport layer stuff
- The current transport layer has been improved significantly but leaves much room for improvement. Ideally we'd like to detect available bandwidth automatically. We have nothing remotely like Path MTU detection; we should automatically adapt to find the right packet size for the connection, both finding what will work at all, and what gives the best bandwidth/loss tradeoff. We tend to get transfer failures on slow connections (not low bandwidth limit, low bandwidth available on that specific connection). We probably should use something like cumulative acks, currently all packets are acked once, it should be possible to ack the same packet twice with 0 bandwidth cost in many cases using ranges. We may want to divide up blocks differently depending on how fast the connection is. We may want to make tradeoffs between fairness to all peers (the current policy) and allowing individual peers more bandwidth for a short period (e.g. because they have requested a bunch of fproxy pages), or have "idle priority" traffic which is only sent when *no* peer wants to send anything (e.g. bloom filter sharing), which may also impact on packet size. And so on. Generally, the transport layer needs to be more robust, especially on slow connections, and it needs to feed information into the load management layer more quickly so that we only accept requests that we can complete in a reasonable time, given the current state of the connection. There are various bugs about this on the bug tracker. Running as well as possible on fairly slow connections is particularly useful in some of the places where Freenet may be needed most.
- Currently Freenet only supports UDP. Make it able to use TCP, HTTP, various steganographic transports (e.g. VoIP). Freenet should provide all the heavy lifting crypto etc, it should be *EASY* to write a transport plugin, just register it with the appropriate type, give block size and so on, and Freenet will do the rest. Last year's work on new packet format should really help although some transports (really small packets e.g. pretending to be Skype) will still need to do their own splitting/reassembly (this should probably happen within the node too, although it should be possible to turn it off).
- Simulating different load management mechanisms would be particularly useful. Simulations of various attacks are also a very important area.
- Different bandwidth limits at different times of the day/week would help many users. We also need support for monthly transfer limits (separate from the existing peak per second limits), and autodetection of the connection's capacity, possibly with options for very latency sensitive behaviour for e.g. gamers (like some Bittorrent clients do). All this must be really easy to use.
- Support for networking protocols with huge delays, for use with packet radio.
- More F2F functionality
Tahrir is a sister project to Freenet, incorporating some of the same ideas, although focussed more on "twitter"-like functionality (or, more generally, microblogging). Much of Tahrir's infrastructure is complete, including a powerful but easy to use low-level transport layer, together with crypto functionality, however the components need to be brought together into a working prototype. More information at The Tahrir Project.
You can sign up as a student until the April 6th deadline here: http://www.google-melange.com/gsoc/homepage/google/gsoc2012