Technology Roadmap Ticket

[ichorid]

After 15 years of development, Tribler has become a running specification of an anonymous torrent client and decentralized content sharing network. However, the path to this specification was an evolution rather than rational design. We changed frameworks, libraries, platforms, paradigms... Now, empowered by the knowledge achieved along the way, we are ready to finally turn this specification into something modern, useful and robust.

This issue holds our technological roadmap. It is not for planning Tribler features, but instead for discussing their implementation.

Tribler component	Current technology	Desired technology	Schedule	Status
Channels backend	SQLite-backed Dispersy	Pony ORM-based Gigachannels	01.07.19	Done
Interpreter	Python2	Python3	01.10.19	Done
Async framework	Twisted	Asyncio	01.11.19	Done
Path handling	os module	Pathlib	01.10.19	Done
IPv8 endpoint	py-ipv8	Rust-backed multithreaded endpoint	01.01.20	In work
Tunnel encryption	libsodium AES-GCM	Wireguard-based ChaCha20 with Poly1305	01.11.20
REST API backend and docs	Hand-crafted Twisted, in-code comments	AIOHTTP, Swagger	01.01.20	Done (AIO)
GUI toolkit	Hand-crafted QT	QML, hand-crafted QT	01.09.20
Multi-path downloads	Single-path downloads	@qstokkink's multi-path downloads	01.03.20
Handling of GigaChannel SQL queries	Channel queries run on reactor thread	A smart queue for offloading SQL queries to external threads	01.03.20	Partially done
Libtorrent wrapper	Twisted-based wrapper	AsyncIO-based wrapper	01.11.19	Done

[qstokkink]

I find myself agreeing with most of the suggested points in the roadmap, but I will challenge the following:

• Rust-backed multithreaded endpoint: why Rust? It seems like Rust is a bad fit for hooking into Python. In my opinion its fine to experiment with, but it will probably never be part of mainline IPv8. If we even want to mess with multiple programming language integration (dangerous!), it would probably be C as it maps more naturally.
• Wireguard-based ChaCha20 with Poly1305 why choose another arbitrary encryption scheme and partition the network even further? ChaCha20 is fine and it works, but I don't see a need for it.
• Quart (async Flask), Swagger I have @DanGraur looking into the best Swagger fit for our project. However, I will note that there are about 200 different Python Swagger implementations, it might not be Quart specifically.

[ichorid]

• I was suspicious about Rust too. However, @dsluijk and @NULLx76 made me change my mind after I did some research on the topic. As it turned out, Rust is super-easy to integrate with Python. Guys basically already done with that. Also, Rust is much safer than C, especially in multi-threaded environments. After all, there are reasons why Rust is gaining so much traction as a system programming language.
• Wireguard-based ChaCha20 is approximately 10 times faster than AES-GCM in terms of CPU load. It is important factor for mobile clients, cloud machines and laptops.

[qstokkink]

Some clarification:

• I'm not doubting that the integration will work. I do want to note that making it work on all our platforms is going to be a challenge (like PyInstaller and Python-for-Android). I do believe the way the integration is being made is sensible. I need to see Rust work (and be significantly faster) to warrant this pain.
• I retract my previous objection to ChaCha20, as it is now available in python-cryptography. This should make it relatively easy to switch.

[egbertbouman]

Correct me if I'm wrong, but I think ChaCha20 is only faster when done purely in software (although 10 times seems like a lot). However, when using a CPU with hardware support (AES-NI for x86, but many other architectures have similar instructions), this may be different. I think this should be investigated first.

[ichorid]

@egbertbouman , I agree, we should investigate that. However, encryption performance will only matter after we move to Rust-based IPv8 endpoint.

[ichorid]

A quick survey on ChaCha20 vs AES-GCM tells us that AES beats chacha flat when aes-ni instructions are available: https://calomel.org/aesni_ssl_performance.html

However, Wireguard page still boasts ChaCha20's advantage over IPSec with AES-GCM:

My "10x" performance statement came from this blog post which compares Wireguard against OpenVPN on a 10Gbit link. Perhaps, the dramatic performance difference it shows should be attributed more to crappy code in OpenVPN, than ChaCha20's performance.

So, ChaCha is ruled out.

[synctext]

Great initiative! Python3 and Libtorrentwrapper have top priority I believe. Then discuss together what should be next.

[hbiyik]

@qstokkink @ichorid @synctext

I can provide some feedback about portability of the source code. I had dealt with cross compiling the codebase for for Python 2.6 and Python2.7 with ARM, x86 android(without python for android) , linux and windows osses in 2017 for Kodi implementation. I can define the experience with one word: AGONY.

https://github.com/hbiyik/script.module.triblercore/issues/2

The problem is not the cross compiling but the method of the 3rd party libraries use when loading the DLLs to python. Some libraries use ctypes runtime loading, some use CFFI runtime loading, some libraries use cython/swig variants, some use pthon.h declerations. Then all the process becaomes a mess.

SUGGESTION:
If a middleware api in CPP to be written to interact with the C/CPP libraries (ie libsodium directly, openssl directly etc, netifaces directly) only for the methods or classes required. And Tribler/ipv8 would only intereact with this middleware in pythonic way. This way the middleware can be ported any platform in an abstract way without much effort.

[ichorid]

@hbiyik , thanks for the info! You're talking about IPv8 Rust-based endpoint, right?

[hbiyik]

@ichorid not only for IPv8 but for all C/CPP libraries for the core (not gui qt stuff, or mathplotlib) , libtorrent, libsodium, netifaces, cryptography backends, (or future lightweight alternative) etc..

[devos50]

@hbiyik thanks for your feedback. Each dependency we add which interact with native code in C/C++ makes it indeed harder for us to port Tribler to platforms with different architectures. This is one of the reasons why we do our best to keep the number of external dependencies as limited as possible. Also, we removed quite a few dependencies when we replaced the Dispersy networking library with IPv8. One of the dependencies of IPv8, M2Crypto, was particularly hard to run on all supported platforms, especially Windows.

Also, now that Tribler is Python3-compatible, it might be easier to port Tribler to Android. The developer team behind the Kivy framework we use to run Python code on Android increased overall stability and usability of the framework. We have some scripts available to deploy IPv8 on Android devices. However, porting Tribler to Android or other platforms is not something we have planned as a short-term goal. since our current focus is on stability and usability. Keeping Tribler stable with all the ongoing (technology) changes is already challenging enough.

[hbiyik]

@devos50 sure thing, just fired another ticket at #4846 to have this ticket clean

[ichorid]

Closing this. Will create a new one when time is right.

[qstokkink]

Cool. We've come a long way.

Regarding the points concerning IPv8 (for anyone stumbling upon this issue in the future), this is how they are continued past this issue:

• IPv8 endpoint: beyond hot-swapping a single endpoint, I have created an issue to make the entire network bindings more loosely coupled (IPv8 v3.0, IPv6 support and network interface refactoring py-ipv8#615). The scope is to easily integrate any new endpoint (IPv6, Bluetooth, a second IPv4 socket, UPnP, .. etc.).
• Multi-path downloads: to make the source code not turn into spaghetti we could also use the network binding refactoring (IPv8 v3.0, IPv6 support and network interface refactoring py-ipv8#615 again) to feed data (cleanly) back into IPv8. That should make maintenance of such a feature at least feasible.

These ideas are definitely not dead! But, not of a very high priority right now.