[Ready to be merged] Peer-to-peer applications

[purplesyringa]

Implements a part of #1425.

Current state

Practically finished, all features are tested.

Docs are ready (check wiki).

[purplesyringa]

IRC test: 1FEL2HhvS48m8SsAPcuyo7irf7J9Ebx449.zip

Create a new site, upload those files, sign&publish and call others to open the site on their computers. Chat without limitations :)

[purplesyringa]

@HelloZeroNet Need your advice.

Currently messages are broadcasted to any ~5 peers (or whatever the developer chooses). But those 5 peers may not have P2P-messages plugin, and they won't be able to broadcast the message. So we need to send the message only to those peers who have a specific plugin.

And here is the question: is it secure enough to add getPluginList FileRequest command? Or is it better to return hashes of plugin names? Or is it better to add hasPlugin? Or anything else?

[Thunder33345]

@imachug is it preferable to have a system ping, system pong
system ping and pong is meant for system(should only be accessed over plugin layer(maybe for more plugins that just connect via general peers unrelated to sites to use))

after you select your possible peers, you send systemping(with noreboradcast) to each peer(s), wait for their ping,
if you receive a pong back you add their ip/id/torid to a list of peers that have the plugin
if they didnt send a direct pong back after time out, it's safe to assume they didnt have the plugin, and add that to a cache for no plugin peers, then go back to step 1 of selecting other peers

ideally the peers will refresh every hour or some interval since things might change
also if you recieve a ping from a peer, you automatically set their IP to have the plugin

[purplesyringa]

Some things not done yet:

• peerBroadcast
• peerSend
• reply to peerSend

[purplesyringa]

Updated IRC

[Thunder33345]

also the IP:
http://127.0.0.1:43110/1FEL2HhvS48m8SsAPcuyo7irf7J9Ebx449 (so theres at least peers)(assuming this is your site)
now, need to figure out how to load this plugin without gitcloning this(done by downloading zip of branch and extracting the plugin https://github.com/imachug/ZeroNet/tree/p2p-msg)

[purplesyringa]

I hope this plugin will be embedded to ZeroNet via this PR, so soon you won't need git clone.

[krixano]

@imachug I've just added it as an option to my installer! Download the new version here.

[purplesyringa]

@krixano Thank you!

[purplesyringa]

A better IRC site for tests:
1DdBX7wzwEaAEuEVQgYpMDhGCmXMXy9W6t.zip

[Thunder33345]

also here's the address for handly sake http://127.0.0.1:43110/1DdBX7wzwEaAEuEVQgYpMDhGCmXMXy9W6t/
rehosting means no peers to chat to anyways
i guess might be time to get the new plugin version

[Thunder33345]

also i keep getting not connected to anyone using that address

[purplesyringa]

@HelloZeroNet If you are by any chance reading this, please reply to #1430 (comment)

[HelloZeroNet]

Probably the most efficient/easiest way would do it using javascript + cors/merger:
If the user has the plugin installed, then add a "p2p channel" site to the client and use that for broadcasting.

[purplesyringa]

Sounds like a good solution without any need to change ZeroNet core. Thank you.

By the way, currently active protection (i.e. validation via JS) is done only if browser is open. However, this may not be the case. We (@Thunder33345, @krixano, @AnthyG and me) decided that we can make a new language, or a JavaScript interpreter which would run some file, e.g. p2p.js from site root when a new message is received. What do you think about this solution?

If you accept it, I would like to separate the plugin to another repository, because with the interpreter it's going to be large. You would either use submodules or subtrees or manually copy content from this another repository. You could either create it under @HelloZeroNet user and make me a collaborator (even if I make some security bug, it won't be added to this repo automatically, because submodules/subtrees aren't updated to the latest version by git automatically) or I can create a repo under @imachug and I'll add you as collaborator -- whatever you choose.

[Thunder33345]

We (@Thunder33345, @krixano, @AnthyG and me) decided that we can make a new language, or a JavaScript interpreter which would run some file, e.g. p2p.js from site root when a new message is received. What do you think about this solution?

i think it should have a seperated plugin, with optional support for p2p plugin
since someone might just want to have something unrelated to p2p be ran at background
a only run on invoked will be better tho since it only exist when a packet is sent, unlike a file that exist until zeronet shutdown in something like nodejs style

[HelloZeroNet]

we can make a new language, or a JavaScript interpreter which would run some file, e.g. p2p.js from site root when a new message is received. What do you think about this solution?

It would be useful in many cases, but I wanted to avoid it as it's hard to make it secure, so I think it's better keep it in a separate repository.

[purplesyringa]

Will you create a repo for me under @HelloZeroNet and make a collaborator or do you let me create a repo under @imachug ?

[purplesyringa]

I'm working on hub stuff. (i.e. read permissions)

[purplesyringa]

Well, if we create our own language, like Solidity, it will be secure. I've created a bunch of languages (and they even worked sigh), so it might be a good idea.

[Thunder33345]

problem of creating a language is not begin flexible enough
if we adopt JS we already have the ecosystem we needed
if we make our new one, that means people have to reinvent things/libraries to get things started

for security sake, we can sandbox each background instance,
we can choose to not allow internet access, and block file system read+write

on the other hand, if we make our own, we dont need to deal with sandboxing, we can just make one that cant escape in the first place, but that means we are few steps behind everyone lacking in basic libraries that programmers rather have

[purplesyringa]

@HelloZeroNet

[HelloZeroNet]

Will you create a repo for me under @HelloZeroNet and make a collaborator or do you let me create a repo under @imachug ?

Sure, but i'm not sure about the name. Maybe "PeerMessage" would be more accurate, than "P2P-Message". What do you think?

[purplesyringa]

PeerMessage sounds good -- renaming.

[purplesyringa]

Renamed

[purplesyringa]

By the way, that stuff running in background should be probably separated to another plugin. I've created repo for it: https://github.com/imachug/ZeroNet-Background

[krixano]

Please also note that the currently requested user actions for installing a plugin are: downloading the plugin code / folder in any way and putting it in a specific folder inside of the ZeroNet one. This already requires very basic abilities and little time.

So what's different from this and having a one-click install function? You still have to trust the plugin and where it came from in both cases.

While I agree that there should be a restricted API for plugins to ZeroNet, I do not agree that the users are incapable of identifying when a plugin is bad and that having a one-click install would be catastrophic. People can flag/comment/vote on plugins with a plugin store zite, we can have a 24/7 running client downloading all plugins from this plugin store and scanning them for malicious code, and users can trust or be wary of plugins by looking at who made them. There are many options to this, but I think the one's I have provided are the most basic and probably easiest, and will increase security quite a bit, imo.

Also, aside from having a restricted plugin API (and perhaps a plugin permission system), I think the security of plugins should be handled by the plugin store rather than the client itself.

[trenta3]

@krixano There was probably a misunderstanding: what I've said is "using the current plugin infrastructure with the additional ability to add a plugin as a one-click action will be catastrophic", so if we had a restricted API for plugins, it would be totally okay to have a one-click install.
The problem really arise when (with current plugin infrastructure) a one-click action could by design execute whatever code on your computer.

I'm particularly interested in the malicious code scan, do you have some references to working POCs or projects for python?

[blurHY]

Permission managing system is also needed for plugins

[purplesyringa]

There is an idea. Not sure whether it works, but if it does, it could simplify installing plugins.

The idea is to have permissions for plugins. I mean, we can have UiWebsocket permission for changing UiWebsocket class, and NOSANDBOX permission for running without sandbox. Users can choose some plugins as safe (@HelloZeroNet's plugins well be marked as safe by default), which is the same as granting NOSANDBOX permission.

The sandbox can be created from my BackgroundProcessing plugin. It turned out to work well for sites (as IP exchange service, backend for KxoID on @krixano's side, etc.) It is not an interpreter of Python, it only makes code that looks dangerous... not dangerous. So it's practically as fast as Vanilla Python.

Moreover, it means that @HelloZeroNet will only have to verify BackgroundProcessing, and other plugins will be executed via it.

[trenta3]

@imachug Good idea to sandbox the plugins in BackgroundProcessing.

However, we should really change plugin architecture, since I don't think it is really modular as it currently stands out (subclassing the existing functions). Maybe having a look at pluggy or something similar?

[HelloZeroNet]

As far as I know there is no reliable sandbox for Python and writing a sandbox in Python itself is considered a bad idea: https://lwn.net/Articles/574215/

[krixano]

It's funny how one of the most used languages for plugins doesn't have proper sandbox support.

[HelloZeroNet]

Is there any language that has proper, supported and well tested sandboxing support? For example Visual Studio Code is written in js, but the plugins are not sandboxed: microsoft/vscode#52116
(maybe we should move this conversation to a Plugin manager issue or similar)

[krixano]

@HelloZeroNet Some languages, like JS, don't necessarily need sandboxing because the APIs etc. are already so restricted. Java has sandboxing built-in, afaik. I believe Lua also has some sandboxing or sandbox-like system.

Additionally, let me expand on what I meant by "restricted API". I mean, the ability for these plugins to access ZeroNet internals should be restricted, not necessarily the OS (although this would be good too if python made this easy). The reason why I make this clear now is because I believe in python, you should be able to restrict access to ZeroNet internals (making things private/protected in classes, modules, etc. - I would think python has these concepts but I don't know much about the OOP side of python).

I feel like I've had this whole conversation before with @imachug

[krixano]

Let me also clear up one more thing about what I was saying before:

• I do think that a restricted API and perhaps permission system, and perhaps (if achievable) sandboxing - would greatly increase security of plugins, as long as it's done right. (Although, whether any of this is actually needed I think is a whole different story.)
• However, I don't think any of that is needed before allowing a one-click install of a plugin
• This one-click install will be called by the zite, the zite giving a description and a zip file to the API command, where the ZeroNet client will prompt the user to install the plugin or not. If accepted, the ZeroNet Client will extract the folder in the zip, place it in the plugins directory, and refresh/reinitialize the plugins. There doesn't need to be any execution on the zite/plugin's side for the installation part (aside from calling the command and providing the zip).
• Security of plugins can be handled by a voting system, plugin review system, plugin scanner, flagging, etc. - which I think should all be implemented for plugin store zites, not the client.

One more thing: I was thinking of sandboxing in the sense of restricting plugins from accessing the OS so that they have to go through a ZeroNet plugin API, but I think actual sandboxing may be different/more complex.

[HelloZeroNet]

Some languages, like JS, don't necessarily need sandboxing because the APIs etc. are already so restricted.

The sandboxing of JS is done by the browsers. For example in a VS Code plugin you can execute an external application with child_process.spawnSync("notepad.exe"). I think if we can't create a secure sandbox that prevent running external applications/creating sockets/etc., then the permission system makes no sense, as any plugin can easily ignore the rules.

[krixano]

@HelloZeroNet I see what you are getting at, although I'm not 100% sure yet.

Additionally, I do think, as I was trying to get across above, that having code scanners, voting system, etc. would have a greater impact in a shorter amount of time.

[trenta3]

Just a little comment to that.
It might be possible to build a python sandbox modifying a little byterun (a good article about it here).

The idea is simply: write a python interpreter inside python itself that allows only non-builtin functions to be called. Every time some interpreted pyton code calls an external module function, the interpreter could use inspect.isbuiltin and raise an exception if it is. If it is not, one should compile the function bytecode and execute that.

This could at least prevent accessing the filesystem, the network and other "external world" functions.

Finer restrictions on possible modifications of the ZeroNet classes I think would require a big amount of work.

[blurHY]

Consider to open an issue to talk about that

[trenta3]

@blurHY I should have done that before. Please use issue #1607 to discuss about plugin system and leave this thread only for things concerning the PR.

[purplesyringa]

@HelloZeroNet What are your plans on this PR?

[filips123]

@HelloZeroNet @imachug This PR is now 1 year old. Can it be merged? Are there any important issues with it?

[purplesyringa]

BTW the PeerMessage plugin repo has been updated, so this PR is outdated a bit. Let me update the submodule real fast

[purplesyringa]

There, done.

[purplesyringa]

I'm closing this PR as new changes should be only made on py3 branch, and there is a py3 version of this PR already.