Google Summer of Code 2021
- Decentralized programming
- Finish porting infocalypse to python3 so it can be used again https://hg.sr.ht/~arnebab/infocalypse/browse
- Improve upon the WoT integration started by operhiem1 https://github.com/ArneBab/plugin-Infocalypse-WebUI
- Content filters for modern codecs would make Freenet viable for realtime streaming
- ogg opus
- webm with vp9 (direct streaming) or av1 (stream on demand)
- Plugin furbishing: Freemail, Keepalive, Freereader, Shoeshop, Floghelper, Flircp, and JFniki need work to make the workflows enjoyable
- https://www.mail-archive.com/devl@freenetproject.org/msg55177.html
- https://www.mail-archive.com/devl@freenetproject.org/msg55180.html
- Packaging
- Add a simple beta-channel for each release by choosing a separate update key with a sitename derived from the next release
- Create a Debian package (most impactful)
- https://www.mail-archive.com/devl@freenetproject.org/msg49740.html
- Create a Guix package (I can mentor that)
- Freenet Mobile https://github.com/freenet-mobile/app
- Mobile node: ref exchange as with https://github.com/ArneBab/Icicle
- Mobile node: spread via NFC and/or Bluetooth
- Spread via Messengers (WhatsApp, Signal, Telegram, Threema, DeltaChat, …)
- Stream-into-Freenet GUI
- Create invite-only friend-communication workflow among darknet friends (mobile first): read + publish as regular, sharing links and chatting via n2ntm
-
REST API for consuming on UI/apps
- See https://github.com/Bombe/fresta
- Add API documentation (essential) with Open API, examples and documentation
- Revamp F2F workflow, see insights here: https://www.reddit.com/r/Freenet/comments/lblvdq/how_would_you_improve_freenet/gm2yrxs
- Revamp Freenet initial page: On first install add help sections and useful links, list official plugins
- Simplify updating and changing dependencies over Freenet: Currently this is a manual, error-prone process which relies on manually patching wrapper-files and blocks at least one pull-request.
This page lists project ideas for Google Summer of Code 2016.
The tasks are split into categories, sorted from high level (web interface, plugins, applications) to low level (crypto, frameworks, transport layer).
- Improve workflows in the web interface
- There is much to do to make it more user friendly. There many small annoyances linked from this bug report. These hurt usability, but the list is not exhaustive: you will find more by simply using the interface and fixing what irks you. A recent but not very detailed, professional mockup focusing on the homepage is here: 1.
An older detailed suggestion might also be an inspiration. There was quite a bit of work on this in 2012 though the project didn’t succeed; it’s worth looking at the devl archives etc.
-
Freenet as decentral, anonymous, community coding platform
operhiem1 addressed the infrastructure parts of this project in Google Summer of Code 2013 by extending the Infocalypse plugin. He simplified sharing code anonymously to: `hg clone freenet:ID/reponame; cd repo; (hack); hg push freenet:MyID/reponame; hg fn-pull-request; hg fn-check-notifications`
But discovery of repositories inside freenet is pretty hard. This is the part which is still missing to create a decentralized, censorship resistant, anonymous GitHub.
Thus as followup, this project would be turning the existing infrastructure into a polished workflow which makes hacking freenet in freenet enjoyable.
Deliverables:
- Add discovery of repositories and connections between repositories by browsing the Web of Trust: “find forks of my repo in freenet”. Offer the option `hg paths – add-forks` (or similar), which shows all public forks in freenet.
- Create freesites with information about the repositories. See the hg site extension for an example.
- Enhance the DVCS web interface plugin to provide nice repository browsing capabilities.
- Add automatic tests for selected collaboration workflows.
- Add communication features, discussion of pull-requests and optionally other parts needed for a full fledged community hacking platform.
- Optionally integrate Infocalypse with the keepalive plugin to improve the longevity of repositories.
-
Website redesign
The current site is working reasonably well, but it is worlds apart from sites like Spacemacs, MediaGoblin or GNU Guile. Experience from the GNU Guile shows the large impact such a change can have. A core requirement of this project is that at the end of the GSoC, the new site must be deployed and enjoyable.
Translation support can be improved - currently it’s not usable with Pontoon, and the download page is currently pretty awful to look at.
- Improve Sone
Sone is a social-networking style system over Freenet. One way to improve it is to make it automatically insert bundles of posts periodically, to avoid having to download all the posts every time. Other problems relate to the question of scalability: When you can’t follow everyone, what does that mean for the user interface? How do you do keyword searches? Etc. There have been ideas on this. Talk to Bombe…
- A good filesharing/file search system
This should tie in with the Web of Trust, allowing users to publish indexes and search those of their anonymous friends, rate others’ indexes, merge them into their own, set up long-term file searches, preload indexes for faster searches, and so on. It might also integrate with Freetalk to help with discussions on labelling or rating. The problems of spam/deliberately corrupt content are very similar on Freenet to on traditional p2p, although the solutions may be different, especially as it isn’t possible to trace spammers; trusted community maintained indexes have developed as a working means of solving these problems on web-based filesharing. Note that we already have a scalable forkable on-freenet btree search system to use as a backend, but it is not yet used for anything, and it is not distributed or WoT-compatible.
- Download from Hash
Integrate filesharing with other networks, using a distributed, WoT-based way to download data by conventional hashes rather than CHKs; this is also related to the wierd stuff (backups) at the bottom.
- Secure reinsert-on-demand
to improve the volume of content that is available. This is a lot harder than it sounds, but in any case we need searching first. (Reinserts using the same keys are seriously insecure, although getting the downloaders to do random inserts may help significantly)
- Keepalive plugin improvements
Some very small changes at the node layer will allow probing for whether a key is available from a random node. Users could then maintain a list of content that they care about; the plugin would download it all as a binary blob, regularly probe for reachability, and reinsert it when needed (possibly single random not fetchable blocks). For this to work really well we might need selective reinsert support in the client layer, but that’s not necessary for a basic implementation. What is important is supporting both files and sites, and letting users publish their lists and subscribe to others. This could even evolve into a full blown distributed backup system, with configurable encryption for files and the ability to recognise which files are popular and avoid inserting them (might need fixing GetCHKOnly).
- A microblogging and/or real-time chat system: Both of these things would actually be implemented in a fairly similar way. Evan has done a fair amount of work on how to efficiently implement microblogging over Freenet. Sone does something like this but is fairly greedy with network resources. Flip does IRC. The fritter-proposal would be a good starting point. ::
- Easy-to-use tools :: We need good, simple tools for inserting freesites (freenet-hosted web sites) and files, and maintaining mirrors of web content.
We already have a blogging tool, but it needs more work, and tools to make it easy to insert existing content etc would also be useful. This should support uploading files of any size, should avoid re-uploading larger files on every update, but should be configurable to do so on a schedule, should work from within the freenet web interface as a plugin, and may support WebDAV uploads direct from authoring software. The ability to mirror stuff from the web would also be useful.
- Better freesite searching
Lots of work has been done on this, but more could be done: Using the new library format, rewriting the indexes on the fly after gathering a few hours’ data rather than writing it from the database over a week, support for long-term searches, web of trust integration, better support for stop-words (maybe aggregating them with common before/after words), tokenisation for tricky languages (Chinese, Japanese), distributing spidering across multiple users (as scaling is getting to be a serious problem now), etc.
- Wiki over Freenet
A widespread wiki over Freenet would be really awesome. There have been attempts, but it is not easy to implement, as there are several challenges. You can learn more there. There have been many attempts; some are hard to use and based on DSCMs, some are easier to use and not scalable.
- More content filters
- We have to "filter" HTML, images, etc to ensure that they are safe for the web browser, and won’t give away the user’s IP address via inline images, scripting etc. Finishing the SVG filter written for 2009, implementing support for SVG embedded in XHTML embedded in ATOM (we have an ATOM filter but it is not integrated yet), maybe an RSS filter. More audio and video formats would be very helpful (particularly WebM and H.264), and with HTML5-based video playback support would make embedded video streaming viable. See here for more on embedded video playback: 2. PDF would be very valuable but the spec is huge, however it is believed that minimal sufficient functionality is not so huge. ODF is similarly a possibility but again is gigantic.
- More simulations
There is lots that can be simulated: Does swapping scale? Do darknet and opennet get on together? In particular, there are a range of attacks on requests that have been proposed, it would be great to quantify them. Load management is another huge area where simulations (and expertise!) would be helpful.
- Tunnels and so on
Somebody with some expertise in tunnels would be very useful. Freenet needs an optional mixnet layer for better security, especially on inserts (where it wouldn’t cost us much performance because we go lots of hops anyway). We have some ideas (something based on Pisces perhaps?)… This is a more research-oriented project but we would like to have some sort of basic tunneling option available by the end of it.
- Much more transport layer stuff
The current transport layer has been improved significantly but leaves much room for improvement. Ideally we’d like to detect available bandwidth automatically. We have nothing remotely like Path MTU detection; we should automatically adapt to find the right packet size for the connection, both finding what will work at all, and what gives the best bandwidth/loss tradeoff. We tend to get transfer failures on slow connections (not low bandwidth limit, low bandwidth available on that specific connection). We probably should use something like cumulative acks, currently all packets are acked once, it should be possible to ack the same packet twice with 0 bandwidth cost in many cases using ranges. We may want to divide up blocks differently depending on how fast the connection is. We may want to make tradeoffs between fairness to all peers (the current policy) and allowing individual peers more bandwidth for a short period (e.g. because they have requested a bunch of fproxy pages), or have "idle priority" traffic which is only sent when no peer wants to send anything (e.g. bloom filter sharing), which may also impact on packet size. And so on. Generally, the transport layer needs to be more robust, especially on slow connections, and it needs to feed information into the load management layer more quickly so that we only accept requests that we can complete in a reasonable time, given the current state of the connection. There are various bugs about this on the bug tracker. Running as well as possible on fairly slow connections is particularly useful in some of the places where Freenet may be needed most.
- Transport plugins
We would like support for stream-based transport plugins, and a bunch of working transport plugins, for both making Freenet work behind nasty firewalls, and for steganography. Some, such as something that looks like WebRTC traffic, will impose particular transport-layer problems due to e.g. packet sizes.
- Bandwidth
Different bandwidth limits at different times of the day/week would help many users. We also need support for monthly transfer limits (separate from the existing peak per second limits), and autodetection of the connection’s capacity, possibly with options for very latency sensitive behaviour for e.g. gamers (like some Bittorrent clients do). All this must be really easy to use.
- Better darknet
Make it easier to invite people to Freenet, provide software bundles, provide a simple password-and-IP-address connect protocol, connect to friend-of-a-friends, make it easy to connect to mutual friends, etc. Lots of ideas about this on the bug tracker. Some of it has been implemented in a branch (foaf-connections).
- More F2F functionality
Make darknet more useful, apart from its obvious role in improving Freenet’s security: Various forms of easy to use chat, possibly realtime, allowing conversations across nodes, both within the web interface using javascript/ajax and via external clients e.g. Jabber/XMPP, possibly with voice support; easy to use, reliable file transfers; selecting downloaded files and bookmarks so that they are visible to your friends of a particular trust level; searching these file lists and transferring the files; possibly automatically fetching files both from friends and freenet; virtual LAN (hamachi style) functionality; social networking style functionality, with very careful privacy protections - after all the friend-to-friend darknet is literally a social network, we should make it as useful as possible without jeopardising privacy.
For some projects in this section, the best solution would simply use an external library to do most/much of the work, and simply integrate it into the current freenet code. So you should do some research into existing solutions first. This principle applies to all projects, but the ones where suitable libraries are most likely to actually exist, are tagged prefer-external.
SSKs currently use our own implementation of DSA. This is old, and the pubkeys are huge. We can switch to ECC, using the Java Cryptography Architecture.
- Make SSKs use ECC for signatures, via JCA (see below).
- Get NSS working semi-automatically. For ECC this is a big performance gain.
- Take advantage of the smaller size of ECC-based SSKs’ pubkeys: optimise SSK requests and inserts (fewer packets etc).
- Take advantage of the smaller size of ECC-based SSKs’ pubkeys: optimise datastore usage (single store for SSKs and pubkeys), and allow keys of ~ 800 bytes (fast, for e.g. IRC over Freenet) and ~ 2KB (slower, for e.g. forums). Automatic, safe migration, even if there are disk space issues, is one of the bigger problems here.
- Implement 32KB SSKs as well, if we’re looking at the datastore.
- Possibly implement Programmable Subspace Keys. The basic principle is to allow to specify a set of rules (in a simple language) that determine whether or not a block is valid for a given key. The simplest application of this is a way to allow multiple posters to use a single SSK (greatly increasing the efficiency of a spam-proof freenet-based chat system), while still requiring them to sign their own posts, allowing the "moderator" to change the key to get rid of a spammer.
There is an official update channel, and while it’s possible to set up alternative update channels, it’s not particularly flexible. It could be good to have support for subscribing to multiple channels: a testing update channel, a stable one, and feature-branch channels. See the (incomplete) update channel RFC. This would include both continuing to refine the design, and implementing initial stages of it, so things like:
- Supporting multiple update channels both in the backend and through the settings UI.
- Build script support for channels - making it easy for an official testing channel or for developers to post custom builds.
- Evaluate or prompt for which build to use given multiple available updates.
freenet uses a custom-written framework for running callbacks and scheduled tasks, which is non-standard and lacks a lot of advanced functionality. Converting everything to use java.util.concurrent and/or com.google.common.util.concurrent would help this a lot. Ideally, the framework would be separated from the rest of freenet, possibly even to the point of being an external library, and provide a clean API for freenet and its plugins to use. This would also allow plugins to do much more advanced tasks (e.g. most of the things in the "plugins and applications" section further below) without having to implement their own framework.
The events framework should be able to, and provide a good API for plugins to:
- run tasks in the background, or according to a particular schedule
- define task+thread matching policies e.g. all database tasks must run in the "database" thread
- cancel running tasks, if the task defines that it’s cancellable
- handle dependencies between tasks, so that e.g. if A depends on B and we cancel/complete B, A is automatically cancelled/completed.
- attach callbacks to events such as task completion/cancellation (e.g. using com.google.common.util.concurrent.ListenableFuture)
- group tasks into related categories so one group doesn’t affect the other. (e.g. tasks for the group "Library/b-tree-write/index-" won’t starve the resources of group "WoT/background-trust-calculation") (this is possibly logically equivalent to the "matching policies" point above but I haven’t thought too much about it)
Currently, config management code is very untidy. Code for read/writing the config in embedded inside the classes they control, which is spread out over many files not related to config management. This creates lots of implicit dependencies, which is not directly apparent in the syntax/structure of the code nor the semantics of the language. (In other words, spaghetti code.)
Config code should be separated from the actual application. It would also be nice if there was an API to expose this to plugins.
The config system should be able to:
- support everything the current config system supports
- use "includes" to separate different sections into different files - e.g. keep vital system-specific settings in a separate read-only file
- read and write values, preserving comments in the file.
-
With regards to mutability, there are three types of values:
- immutable
- mutable (takes effect after a restart)
- mutable (effects the running daemon immediately)
Note that the actual code for doing the setting should be implemented outside of the config system.
-
See the application template on the GSoC website.