Roadmap

Stephen Oliver edited this page Oct 31, 2016 · 9 revisions
Clone this wiki locally

Table of Contents

0.8

Features that we cannot possibly release without

Fix the installers
The windows installer often breaks and we currently need a 32 bit Java for the wrapper.

Features that will probably go in

Fix or remove Library
the new format indexes are easy to update and up to date, but the search usually fails. The code has been updated to keep the metadata for each btree node in the node above, we simply need somebody to run the spider on a fast machine to rebuild such an index. Hopefully performance will then be reasonable. Alternatively just remove search for the time being and depend on the index sites and bookmarks. In 1474: Replaced outdated indexes, not activating Library by default.
Darknet enhancements
See 0.8.5. The easy parts from this would be very useful, given we want Freenet to grow virally as a darknet. Hopefully this will help with marketing in general, especially as we are having to show fairly strong caveats for opennet in the first time setup wizard. Parts of the darknet enhancements have already been implemented. Specifically, we want to implement most stuff related to FOAF connections and invites.

Already implemented

Fixed wininstaller
The Windows installer must work reliably. The alpha has been deployed. The main difficulty is that the current code cannot shut down the installer reliably, so it has to be done by hand in updating, and uninstall sometimes doesn't work immediately. We need to fix this before 0.8.0. However, the service code has been gotten rid of, so install mostly works. There are also issues with virus checkers; some of them have whitelisting programmes.
More peers for faster opennet nodes.
Maximum number of connections is variable from 10 to 140, based on output bandwidth.
Plugin updating over Freenet.
Bulk vs realtime flag for requests
Bulk requests are optimised for throughput, realtime requests optimised for latency. Fproxy uses realtime requests, big downloads would generally use bulk requests.
Datastore I/O reduction.
Specialised structure replacing bloom filters for the store: keeps a map of the first few bits of the key. This is implemented in the store-io branch which has not yet been deployed.
Freemail
Zidel improved Freetalk to integrate with the Web of Trust, with shorter addresses, using CAPTCHAs to prevent spam, better performance and reliability, and have a web interface, for his Google Summer of Code project.
Removed db4o
During summer 2014 toad ripped out db4o. This was deployed as purge-db4o in release 1468, with an urgent bugfix in 1469. This will need to be mandatory and there may still be rough edges to check on.
Multi-container freesites
Thanks saces. Finally merged for persistent requests in purge-db4o.
Make wrapper.jar updateable.
This is implemented and apparently working in most cases (very old installs may have problems). Hence we have fixed some nasty bugs by updating the wrapper. It is planned eventually to split up freenet-ext.jar more fully.
fixed MP3 filter + audio tag.
Optimisations to CPU, disk I/O
Auto-update improvements
Make it easier to change the auto-update keys should that be necessary. Also some small changes to plugin updating to enable us to update essential plugins, and avoid forcing non-essential updates immediately, i.e. use the old one and download the new one, deploy it after asking the user or after rebooting.
Easy way to insert big freesites
Provided by jSite and freesitemgr (from pyFreenet) for sites up to several hundred files and several GiB of data.

Features we could merge

Ogg Vorbis, Ogg Theora
Sajack's branches support filtering these common, open audio/video formats. These will be merged before 0.8.0.
RSS, ATOM and SVG filters
Requires a new freenet-ext.jar with the JDOM library, but the code already exists. It may need testing though. The CSS filter by the same author required a lot of work to get polished and tested and working properly.
Sone
The popular microblogging/social chat app would be worth making official. It has scalability problems but the solutions are fairly clear and can be dealt with later.
Jfniki
Jfniki is probably worth making official, provided it is in an appropriate form (using toadlets).

Features that would be useful and aren't too difficult

0.8.5

Proposed point release focusing on darknet: security, usability, functionality. No doubt many small unrelated things will creep in too.

Likely features: DARKNET!

Fix the Pitch Black attack
This is essential if there is going to be any serious use of darknet. Oskar has a proposal but nobody has simulated it yet. See bug 3919. Might be worth trying to get something published once this is sorted out.
Connect to FOAFs
This could dramatically improve performance on darknet especially for newbies, making it worthwhile to get on freenet via a single friend.
Proper invites
Probably including an installer, one-time tokens which can be used to get connected. Probably would include several friends to try to avoid NAT and uptime problems.
Option to see friends of a friend and upgrade them to friends if you know them
Obviously all these things would be optional (overall opt-out and have a visibility level for each friend), but this is exactly how "conventional" social networks grow. This could help a lot. We would also want to tell the user when his friends add new friends - provided they are public. Upgrading a friend of a friend to a friend would probably require out of band password verification for security.
Share downloads and bookmarks persistently and good user interface for friends, friends of a friend, files, bookmarks etc.
Maybe more like social networking interface. Show name, connection status, recent direct messages, bookmarks, files, etc, status message if any etc... Some sort of summary on the friends page, detail with all the files etc on one or more separate pages. Mark each of our downloads, uploads and bookmarks with a trust level so that we can share them if we want to. Provide the ability to search our friends' public downloads.
Fast downloads from friends
Once we've found a file in a friend's downloads, we should be able to download it both from Freenet and from that friend, and from other friends too. This will be on a different level to bloom filter sharing as it probably won't be in their datastores.
Bloom filter sharing (darknet only)
This will very likely lead to serious exploits on opennet, but is quite viable on darknet. There are some questions over exactly how to implement it of course - sharing the new format slot filters might allow censorship attacks for instance, so we might need to keep, or generate regularly, chopped up bloom filters.
Much better friend to friend chat
Fairly important IMHO. Maybe social network like - private messages plus public forum/wall etc.

Postponed features from 0.8

Clean up freenet-ext.jar
This should be split up more fully. infinity0 has done some work on this. We already have the infrastructure to deploy and update separate jar files for different components. Taking full advantage of this will allow us to deploy code depending on more recent jars, such as some of the filters that haven't been merged yet. Solving this properly will require finding a general purpose solution for how to safely build code that uses Maven.

0.9

Likely features

New user interface
Ian is very keen on this. It will make extensive use of Javascript and may or may not require it, this is still under debate. It is likely a large enough project that it will be a 0.9 feature rather than 0.8.
Bloom filter sharing
Should improve performance significantly on darknet, and also provide a way to soak up extra bandwidth on fast connections, reducing the possibilities for traffic analysis. Whether it will be significant on opennet is not yet clear: It is possible to exchange bloom filters with long-term opennet peers, it is pointless if they are not long term however...
Web-pushing
Sashee's Summer of Code project, which features dynamic updating of parts of the UI and which loads all inline images at once and shows progress for each (thus solving the browser connection limit problem), is merged but disabled by default because of a design flaw resulting in losing events and thus stalling (especially on slow browsers). It would be a very good idea to fix this and get it on by default. It will remain optional.
Keepalive related stuff
The Keepalive plugin supports monitoring files' availability and reinserting them. We need better integration with this. In particular, we should keep the top block as a binary blob, so that SSK-based splitfiles can be reinserted easily. Also, it should be possible to trigger Keepalive or similar (reinserts etc) from either the download or upload page. Further, we probably need some sort of probe request to allow checking for availability while bypassing nearby caches.
Time-based protection
A relatively small insert could be sent with a flag indicating not to route the request to any node added after a particular time (the limiting factor would be how long ago that time was). This would beat mobile attacker source tracing for reinserts (although of course it remains possible between inserts). The real solution, either rendezvous tunnels or secure bursting (broadcast all requests at once and then stream the data afterwards, requires capacity based load management, probably only works on darknet), will be implemented in 0.10.
More work on Library
The Library search plugin needs to be reliable and very fast. It also needs to deal with stop words (and in e.g. Chinese, stop characters) efficiently. There are various options for this on the bug tracker. Most notably, prefetching the top layer or two, aggregating stop words/chars with adjacent words, and structuring the on-freenet format so that we only need to fetch a whole node if the fetch of the single block we need fails (although we will fetch some of the other blocks anyway for redundancy), which should mean we can do a search in approximately one block per layer.
Integrate Freemail
We need better integrated Freemail, so you can send a private reply to a public message.
Fix the Pitch Black attack
This should have already been fixed.
Filesharing tools
Once Library is working well, basic distributed file search based on WoT would not be all that difficult. This is worth seriously considering.
Multi-Hash Keys
Redundancy in the top block. Splitfiles (any file that doesn't fit in 32KB compressed) have 100% redundancy - we insert N data blocks, and then another N check blocks, and within a segment (128 blocks), we can reconstruct all the blocks from any 128. But above all this there is usually a single CHK block. While this is probably more popular than the lower blocks, there is still a very real possibility of it getting stuck in a black hole, or being hard to fetch after a while: we initially push data to 24 nodes' caches, but it will only persist in the store on 3 nodes, and they might be offline. However, there is data to suggest that improvements to inserts can get the same effect but without the disruptive new key type.
Pause mode
and a system tray icon to trigger it. This is popular on uservoice but given the fast bootstrapping we are now seeing on opennet it probably isn't necessary.
More updater changes and more secure deployment
Various ideas for this on the bug tracker e.g. separate the keys needed to activate a release from the actual SSK to insert it.
Freetalk
Freetalk is now an official plugin, which can be loaded easily. It will eventually be bundled, with the user encouraged to create an identity during or shortly after install. Freetalk requires significant work on optimisations as well as some usability work. Mostly these rest relate to CPU, database and disk usage of Freetalk, in particular WoT and Freetalk need to communicate in an event-driven manner (currently it does a lot of unnecessary database access and can lose identities and then get them back, which is very bad for usability).
IMHO (Toad), it will be necessary to buffer writes in such a way that transactions can be aggregated while maintaining consistency and the ability to rollback, via a custom IoAdapter. What could be a lot more serious is the effect on the wider Freenet of a lot of WebOfTrust/Freetalk users (a significantly larger userbase than FMS, which we can reasonably expect if we bundle Freetalk). RecentlyFailed request quenching, and new packet format, have been implemented and should help to reduce the overhead of polling. Hopefully we won't need full passive requests just yet.

More content filters

More audio/video filters
We need more content filters. Matroska/VP8 would be a good start. Unfortunately getting the documentation for H.264 is rather expensive, so we will probably stick with open formats for now, but VP8 is a huge improvement over Theora. Many organisations inserting video are able to re-encode it without too much trouble. Also, sajack's work includes the beginnings of a playback UI initially capable of streaming and which could be extended to play back only those parts we've downloaded i.e. skip segments that didn't download. We also need a filter for FLAC.
Other filters
PDF files are really important for e.g. leaked documents. They are also potentially compromising on many different levels. A basic PDF filter sufficient for most purposes could probably be implemented relatively easily, although full support would be a fairly large project (the spec is monstrous!). OpenOffice files and so on might be a possibility but would be a big job. Javascript filtering is possible in theory, see the jobs page, but would be a huge task and not without residual risk.

0.10

Likely features

Passive requests
Essential in the long term for scalable chat applications, but also a big enhancement in themselves.
Long-term requests
An extension of passive requests, allowing for data to move while the originator is offline. Strongly dependant on capacity based load management. Big step towards HardStego i.e. non-real-time transports.
Capacity-based load management
New load management in 0.8 goes some way towards this. But for it to really shine, and to enable long term requests, we need to have, on darknet connections, a very large capacity for long-term requests.
Secure bursting
Send all the requests (or inserts) at once, and they are routed immediately (and passed around efficiently, often in a single message). Then the data is transferred over a longer period. Timeouts occur on the overall structure not the individual requests. Hence we beat mobile-attacker source tracing even if we are doing large predictable (re)inserts.
Rendezvous tunnels
Encrypted tunnels, quite expensive but not strictly source routed, but provide an anonymity set comparable to the whole network, used by paranoid users who don't trust their peers and for the "top blocks" i.e. the predictable part of an insert.
Transport plugins
Worth seriously considering, very popular on uservoice. No real security gain except on darknet though, and probably quite a bit of work.

1.0

Important work

Lots of debugging!
Consider a proper, separated, clean, stable plugins API.
Lots of work on the UI.
More content filters if possible.

Post-1.0

Possible features

Plugin sandboxing
This is probably post-1.0.
Javascript filtering
This is probably post-1.0 - or never (given that securing Javascript is an open research question).
Long-term requests with offline support
This *might* be post-1.0. It could build upon the shoeshop plugin.

See also