Rebase ZeroNet infrastructure to IPFS or Dat

[0polar]

ZeroTalk: http://127.0.0.1:43110/Talk.ZeroNetwork.bit/?Topic:1538407463_14ytAKDfNjArMTqGecTi7ginG3aZTsRAum/Rebase+ZeroNet+infrastructure+to+IPFS+or+Dat

Advantage: DHT, archiving system, bigger developer team & community, no repeated code…
Also a chance to migrate to Python 3.

Before I start to write code, is there any suggestion for me?

[0polar]

I want ZeroNet focus on distributed web apps framework, and unite other distributed storage infrastructures together.

So finally network architecture will be like this:

• distributed web apps
• ZeroNet (distributed web apps framework)
• BitTorrent used by ZeroNet / Freenet / GNUnet / IPFS / Dat... (distributed storage)
• I2P (anonymous overlay layer)
• cjdns (replace IP/BGP)

And mature infrastructures have more code review & security audits. ZeroNet security is considered bad.

[ghost]

CJDNS requires IPv4/6 connectivity and regardless of the prefix composition MP-/BGP is going nowhere.

[danimesq]

CJDNS doesn't requires IPv6 to work, but ZeroNet needs IPv6 (already being implemented in a pull request) to run thought CJDNS

[szh7379]

https://geti2p.net/en/blog/post/2018/10/23/application-development-basics

I think the i2p team do that for zeronet. it is python.

[danimesq]

IPFS uses too much system resources

[0polar]

IPFS seems too bloated and hard to use. Dat looks good.

[danimesq]

@0polar

DHT, archiving system, bigger developer team & community, no repeated code…
Also a chance to migrate to Python 3.

Before I start to write code, is there any suggestion for me?

DHT is really need, and already used on IPFS and Dat.
@HelloZeroNet @shortcutme could you cooperate with him on coding that? ZeroNet's network haves support to merging it with IPFS and Dat?

I want ZeroNet focus on distributed web apps framework, and unite other distributed storage infrastructures together.

So finally network architecture will be like this:

• distributed web apps
• ZeroNet (distributed web apps framework)
• BitTorrent used by ZeroNet / Freenet / GNUnet / IPFS / Dat... (distributed storage)
• I2P (anonymous overlay layer)
• cjdns (replace IP/BGP)

I don't think ZeroNet is a framework for distributed web apps, but another network for that.
Regarding i2P, Tamas Kocsis says it is hard to implement, but it isn't so: Vuze torrent already implemented that for both inbound and outbound connections. So ZeroMux was right that the ZeroNet creator is lazy to implement i2P (even the IPv6 pull request is yet open without a review). Regarding Python 3 planned to end year of 2018, I understand Tamas, as he haves no obligation to follow the roadmap, as the donations aren't even used.

And mature infrastructures have more code review & security audits. ZeroNet security is considered bad.

Now I understand Tamas. He is alone building a huge project, and isn't handling well the donations he received. The administration of he is really bad, but we shouldn't forget he is a human and probably with a good will. The only things we should blame he is about creating an organization to store ZeroNet-related repos, opening this organization and ZeroNet core repo for new maintainers, and distributing the ZeroNet donations to those maintainers and making a collaborative roadmap.

Regarding security, openned that issue: #1849

@szh7379

https://geti2p.net/en/blog/post/2018/10/23/application-development-basics

I think the i2p team do that for zeronet. it is python.

Strad tried to help implementing i2P to ZeroNet, but was ignored. Tamas keep saying that it is difficulty, but doesn't haves dedication to that. Vuze is the proof that BitTorrent networks (as ZeroNet) can support full i2P: https://wiki.vuze.com/w/I2PHelper_HowTo

IPFS seems too bloated and hard to use. Dat looks good.

Yes, but IPFS is pretty modular, which futerely will support to storing bytes on merkle tree instead of full files. On IPFS thats really bad to re-upload a whole file for just changing an comma. Dat seems to not work at this manner, being more lightweight.
Both IPFS and Dat haves their purposes, while ZeroNet's purpose is to have some cool web features and built-in Tor. But as network, I think ZeroNet is just redundant, so it is better to use another network such as Dat or IPFS. ZeroNet should be a good interface than an network, to power decentralized sites with features as site clonning.

[danimesq]

Asking what those ZeroNet and other network's members thinks about that:
ZeroNet: @imachug @April93 @krixano @anoadragon453 @trenta3 @rllola @tangdou1 @slrslr @Thunder33345 @blurHY @krzysztof113
i2P: @str4d
Dat: @pfrazee @mafintosh

I've also suggested a translator plugin (#1835), to permit zites sending commands that are repassed to the main frame (like Cordova, Electron and Ethereum commands).

[0polar]

I am afraid ZeroNet API need blow up and rewrite, too.

• <iframe> as sandbox are too hacky
• WebSocket as API are difficult to use (compare with HTTP REST)
• JSON as database are not look like tables (compare with CSV)

[danimesq]

I am afraid ZeroNet API need blow up and rewrite, too.

I agree.

<iframe> as sandbox are too hacky

Dat-Fox (an Firefox extension for Dat) used that solution for not using iframe:

Dat site operate on the correct origin. When using the dat-gateway to bridge to the dat network, all dat addresses look like http://localhost:3000/{hash}/path. This has the effect of potentially breaking relative URLs on the page, and also preventing the web's cross-origin policies from preventing data leakage between sites. To fix this we have to make {hash} the origin.

See: https://github.com/sammacbeth/dat-fox/#dat-firefox

WebSocket as API are difficult to use (compare with HTTP REST)

Getting rid of <iframe> will fix that. Or else, an Translator/ScriptEcho plugin for bringing secure code to main frame should help: #1835

JSON as database are not look like tables (compare with CSV)

JSON is the most modern web standard; for easy editing, there could be an SQL viewer that supports that dbschema. Although CSV could also be used.

[0polar]

Yes, but IPFS is pretty modular, which futerely will support to storing bytes on merkle tree instead of full files.

Here is Dat docs: https://docs.datproject.org/faq#dat-vs

• Dat has built-in archiving and versioning, IPFS have no
• Dat has built-in synchronization, IPFS have no
• Dat already supports exchange the deltas to complete a full sync

[0polar]

To fix this we have to make {hash} the origin.

My plan:

When user visit http://{hash}.zeronet in browser, WebExtension intercept this request and proxy to http://localhost:43110.

ZeroNet client should check the Host HTTP header.

.zeronet is not a vaild TLD, so there is no conflict.

[danimesq]

In order for IPFS to provide guarantees about interoperability, IPFS applications must use only the IPFS network stack. In contrast, Dat is only an application protocol and is agnostic to which network protocols (transports and naming systems) are used.

How could be used on ZeroNet?

[0polar]

How could be used on ZeroNet?

ipfs-inactive/faq#119 (comment)

[danimesq]

How personally you think ZeroNet could be a middle network for Dat?

[anoadragon453]

WebSocket as API are difficult to use (compare with HTTP REST)

Fwiw I just worked with implementing ZeroFrame in a separate language by communicating with ZeroNet via WebSockets and besides needing a quick PR for an IP address matching issue, it was quick easy to work with. You just need to open one connection and you can talk and stream data. And it's fast.

Dat's proposal for an iframe replacement is interesting, I'll have to look into exactly how much separation that provides.

[danimesq]

Dat's proposal for an iframe replacement is interesting, I'll have to look into exactly how much separation that provides.

I'm sure most of ZeroNet users are in favor of replacing it; iframe is the most serious and retrograde limitation of ZeroNet.

iframe needs to replace existing code (such as LocalStorage) to the ZeroFrame's one, zites doesn't supports iframe embeds, haves bugs when opening links onto a new tab, needs to add expensive codes to make it working with base and relative/hash links, etc.

[blurHY]

So how can we make a sandbox then
Disable js,like tor ? impossible

[danimesq]

@blurHY

So how can we make a sandbox then
Disable js,like tor ? impossible

Common deep-web onion sites that runs thought Tor doesn't haves an iframe.
Also, Tor is so retrograde as the ZeroNet iframe.

[blurHY]

Security is important.

[danimesq]

Security is important.

Then move to i2P.
That's what I've suggested lots of times for ZeroNet's creator.

Regarding Tor-wise use, .onion sites are considered secure and doesn't uses iframe, so Tor bundle browser takes care of disabling unsecure JS.

[blurHY]

OK.We can suggest shortcutme add an option to disable iframe for the sites you trust

[danimesq]

OK.We can suggest shortcutme add an option to disable iframe for the sites you trust

@shortcutme

How ZeroNet would be more secure than Dat-Fox by using an iframe?

[HelloZeroNet]

It would solve some cross-site security problems, but notifications, sidebar, loading screen would not be possible in secure way without the iframe

Other problems:

• browsers automatically turns addresses to lower case, so it would not be possible using the current address format
• need to install extension to every browser (another trusted third party with all your data)

Btw there is a similar chrome plugin for zeronet: https://github.com/goldenratio/zeronet-protocol-crx

OK.We can suggest shortcutme add an option to disable iframe for the sites you trust

You can request NOSANDBOX permission which disables it for the site

[danimesq]

You can request NOSANDBOX permission which disables it for the site

Decentralized sites should be trustless; the contrary of when using an NOSANDBOX permission.

It [getting rid of iframe] would solve some cross-site security problems

Good

but notifications, sidebar, loading screen would not be possible in secure way without the iframe

Notifications can be shown from the ZeroNet app itself, also the suggested desktop notifications from feed.
Sidebar can work on ZeroHello, with a dropdown list to pick an site, also clicking on an site on site list will show its sidebar.
Loading screen can come from the ZeroNet app as a little window on the bottom-right, showing which sites are currently being downloaded and how.
Also, an dedicated ZeroNet browser like Beaker could have all that.

All these suggestions seems to be hard, but if you thought on they since the begin of ZeroNet, would realize that all these are better and less hard than implementing a whole nonce/sandbox/iframe system.

Other problems:

• browsers automatically turns addresses to lower case, so it would not be possible using the current address format
• need to install extension to every browser (another trusted third party with all your data)

The second problem is related to the first one.
About the first, it would really need an companion extension, or using case-sensitive lowercase hash that is converted to Bitcoin address/Dat hash.

[danimesq]

@pfrazee
What do you think about ZeroNet using Dat?

[0polar]

So how can we make a sandbox then
Disable js,like tor ? impossible

Prerequisite knowledge for you:

1. Every pages are already have sandboxed.

2. If pages are in same origin, they can interact each other, otherwise they can't.

3. Browser think http://{hash1}.zeronet http://{hash2}.zeronet are not same origin.
   This is proper.

4. Browser think http://localhost:3000/{hash1} http://localhost:3000/{hash1} are same origin.
   This is wrong. So we need workaround.

[0polar]

notifications, sidebar, loading screen

Put these in WebExtension

browsers automatically turns addresses to lower case, so it would not be possible using the current address format

Yes, this is a problem for ZeroNet.
Dat have no such problem, because Dat official URL format are hexadecimal.

[0polar]

How personally you think ZeroNet could be a middle network for Dat?

middle network?
I mean, Dat become the storage layer of ZeroNet, ZeroNet should focus on application layer.

[danimesq]

Put these in WebExtension

Good, and also this could be on the app itself.
IPFS haves a very good browser companion extension, and ZeroNet should have also an extension with more than the zeronet-protocol-crx haves.
@goldenratio, would you like to improve it?

Yes, this is a problem for ZeroNet.
Dat have no such problem, because Dat official URL format are hexadecimal.

Using Bitcoin address haves the only advantage of accepting donations for the same site address.
But Dat hashes/addresses are very better, and ZeroNet is getting deprecated on not accepting it; using Dat protocol altogether with the amazing ZeroNet page/user features will take ZeroNet to another level of use and engagement.

middle network?
I mean, Dat become the storage layer of ZeroNet, ZeroNet should focus on application layer.

Good.
But I doubt @HelloZeroNet @shortcutme (I don't like to use two mention for the same person, HelloZeroNet should be moved to an organization instead of personal profile) will accept that; but the ZeroNet health is in question.
Using Dat and getting rid of iframe (I didn't thought about that on the past year) are the most important thing for the surviving of both ZeroNet and Dat protocol: one depend on other while the rise of IPFS.

@pfrazee
You've liked to ignore, but I kindly ask collaboration between Dat and ZeroNet for a more open web, despite our previous misunderstandings.

As IPFS haves its own stack, I agree with @0polar that ZeroNet should be focused as application layer (as, I repeat, it haves amazing, creative and exclusive page/user-wise features), while Dat is its storage layer. Dat will be very helpful for ZeroNet, the same way ZeroNet will be helpful for Dat.
About replacing the current storage layer of ZeroNet:

• When clicking "Create new, empty site", it will open an modal dialog, showing if user wants to pick an existing project or create a new folder, and where to save that folder (supporting custom folders). This will solution the need of using vanitygen to make folders more easy find, like Beaker browser already does.
• When creating the new site, instead of assigning an Bitcoin address to it, an Dat hash will be created, and its privatekey saved (like Bitcoin address are)
• When signing, an commit (snapshot) will be created using Dat
• When clicking publish, it will be published to Dat network
• User can access zites with Dat
• An new API/command, were an site that is using Bitcoin address, will show an notification that is available on Dat, with link for redirection

If there are already Dat libs for Python, that will not be very hard to implement.
User certs can still use Bitcoin address, as it will be used just for signing.

[blurHY]

OK.I agree.
Just build a browser based on Firefox for zeronet (Tor Browser is based on Firefox,so firefox may be better).
What time to add it to todo ?

[Thunder33345]

You can request NOSANDBOX permission which disables it for the site

yes we can, but it's not ideal and dose not give fine grained control, nosandbox works but it also gives control over other sites
i think this is fundamentally fault on using a localhost as proxy approach
and i dont see any way to avoid it other then to restructure which probably would be too late

[0polar]

Fix your English:
localhost as proxy -> localhost as user interface
restructure -> refactor

[blurHY]

as user interface

local host is not user interface.It's web server

[0polar]

local host is not user interface.It's web server

They are not conflict.
http://localhost:43110 is ZeroNet web server for user interface.

[blurHY]

Fix your english:
They don't conflict

[HelloZeroNet]

Dat works differently and has different style database. So if you want to develop for dat efficiently, then you should use their API and no real reason to interact with zeronet-style api.

[blurHY]

@shortcutme Is it possible to remove iframe by building a browser for zeronet

[HelloZeroNet]

Sure, it's possible.

[blurHY]

So why not build a browser as the user interface of zeronet.

[0polar]

So why not build a browser as the user interface of zeronet.

Understanding browser's source code is a hard task and no necessary.
WebExtension plan is good enough.

[blurHY]

Using extension may be not enough yet

[0polar]

then you should use their API and no real reason to interact with zeronet-style api.

Yes, but if do this, there is no compatibility.
And I think complete rewriting will better.

[danimesq]

@blurHY

OK.I agree.
Just build a browser based on Firefox for zeronet.
What time to add it to todo ?

Why fork Firefox if we can create an plugin for it and support the ZeroNet Browser by @rllola ?

@0polar

Fix your English:
localhost as proxy -> localhost as user interface
restructure -> refactor

blurHY

Fix your english:
They don't conflict

Please don't start an english discussion, but focus on what matters here.

@shortcutme @HelloZeroNet

Dat works differently and has different style database. So if you want to develop for dat efficiently, then you should use their API and no real reason to interact with zeronet-style api.

In which thing Dat works differently: because it isn't using Bitcoin addresses? because it is fundamentally better than IPFS and better than ZeroNet's network/storage layer (but Beaker is not better than ZeroNet on application layer)? because (like I've reported about Zero Games, which proves ZeroNet is not really censorship-resistant, which was a shame) Dat is really against censorship and for an open Internet (unlike ZeroNet that relies on an censurable Bitcoin address)? because Dat haves built-in archiving and versioning? because Dat is being supported by Mozilla and added support to their protocol on Firefox? Can we point more about how ZeroNet is dying as a network and storage layer, and wait for more zites being censored through Tor and the unversioned/confuse updates of ZeroNet? And if an attacker does worse than what occurred on ZeroGames and, instead of replacing index.html, putting chldporn files to give legal issues to their seeders? That's the health of ZeroNet what is in question. Or I support Dat (and we've recently had serious disagreements, but they were instructed to do that about me) or I support ZeroNet; the two networks needs to be united or only one be used. And I'm not wrong about that, because I'm sure users doesn't likes duplication. That is not a corporatism competition, where Dat/ZeroNet could think "I want money, you make money, I'm competitor and competition is good for consumers they haves options". Users prefers an product that supplies their needs, instead of opening Dat and ZeroNet at same time and being confuse. And competing to Dat, death for ZeroNet as a network. But ZeroNet is better at application layer, but its network will kill it. Dat already haves an microblogging with user features (Fritter), and when it gets more application-layer updates, there will be no reason for using ZeroNet, as Tor can be used through Dat. ZeroNet should do an huge and unnecessary work for rebuilding a network from scratch, while Dat (through Beaker) haves possibility to implement application-layer features: so ZeroNet is on disadvantage.
Dat needs an strong application-layer, and ZeroNet needs an strong storage-layer: so @pfrazee and @shortcutme , if you two really fukng cares about an open and decentralized Internet, both you should unite your efforts as each one of you haves the piece of the other side, that jointing will form an very good application/storage-layer stack that is better than IPFS (the focus is not be better than other, but to be better for users).

blurHY

@shortcutme Is it possible to remove iframe by building a browser for zeronet

shortcutme

Sure, it's possible.

An browser for ZeroNet is already being made, with Python. Maybe he wants to contribute: https://github.com/rllola/ZeronetBrowser
Also, removing iframe through an custom browser or browser extension, is ok.

0polar

Understanding browser's source code is a hard task and no necessary.
WebExtension plan is good enough.

@rllola is already building an browser for ZeroNet.
WebExtension is good, so @goldenratio could improve his extension.

blurHY

Using extension may be not enough yet

I agree.

shortcutme

then you should use their API and no real reason to interact with zeronet-style api.

0polar

Yes, but if do this, there is no compatibility.
And I think complete rewriting will better.

Tamas simply doesn't wants to do that. He can't think how that can be done, even if we explain how and why it matters.
ZeroNet user logins and other application-layer features can work through Dat.
Dat is just a replacement for Bitcoin addresses and current network overall, while the application-layer is kept the same, and will benefit Dat while that will benefit ZeroNet.

[blurHY]

So, we should focus on app layer and let dat's team to do other things.
That's what stack means

[HelloZeroNet]

I would like to keep ZeroNet as independent as possible, so the custom zeronet network won't be dropped.
Support of downloading ipfs/dat/torrent files could be possible with custom, third party plugin. Rebasing ZeroNet to dat is not possible as it's written in different language and the protocol is also have differences (not just the network addresses).
Rewriting from scratch is possible, but it won't be backward compatible because of the differences so probably it's better to find a new name for that network instead of zeronet. If you planning to do that I suggest to open a new repository.

[blurHY]

@shortcutme Is plugin system planned ?

[anoadragon453]

@blurHY A plugin system is already implemented and even used extensively for core functionality.

https://github.com/HelloZeroNet/ZeroNet/tree/master/plugins

[anoadragon453]

This can already be done on sites with NOSANDBOX, but it would be cool to see in ZeroHello, yeah. Topic for a separate issue though.

I recommend closing this one at this point as @HelloZeroNet has stated he will not rebase ZeroNet on top of another incompatible network.

[0polar]

Is plugin system planned ?

Off-topic.
I suggest you start another thread.

[HelloZeroNet]

Yes, it's planned in some careful way as it allows direct code execution. It would be nice to find a way to sandbox zeronet inside a directory, so the worst thing could happen is leak the private keys.

[anoadragon453]

It would be nice to find a way to sandbox zeronet inside a directory, so the worst thing could happen is leak the private keys.

Doesn't everything pass through the WebSocket API anyways? Can we not do path-based rules there?

[danimesq]

@blurHY

So, we should focus on app layer and let dat's team to do other things.
That's what stack means

Totally agreed.

@shortcutme

I would like to keep ZeroNet as independent as possible, so the custom zeronet network won't be dropped.
Support of downloading ipfs/dat/torrent files could be possible with custom, third party plugin. Rebasing ZeroNet to dat is not possible as it's written in different language and the protocol is also have differences (not just the network addresses).
Rewriting from scratch is possible, but it won't be backward compatible because of the differences so probably it's better to find a new name for that network instead of zeronet. If you planning to do that I suggest to open a new repository.

"Rewriting from scratch is possible, but it won't be backward compatible because of the differences so probably it's better to find a new name for that network instead of zeronet. If you planning to do that I suggest to open a new repository"

Ok, we should support Dat instead.

[0polar]

When user visit http://{hash}.zeronet in browser, WebExtension intercept this request and proxy to http://localhost:43110.

Update:

No, WebExtension is not necessary.
Just like how people resolving .i2p and .onion addresses, a normal SOCKS5 proxy can do this enough.
https://stackoverflow.com/questions/33099569/how-does-sock-5-proxy-ing-of-dns-work-in-browsers

[krzysztof113]

Hi
Do we have a standard of urls, protocols format (beside http://127.0.0.1:43110/hash) that is part of ZN? Like mentioned above http://hash.zeronet. I see a lot of different approaches. I.e. http://localhost:port/hash, zn://hash, zeronet://hash etc.

[blurHY]

No standards. It depends on implementations.
For ZeroNet, there's only http(s)://address:port/publickey

[beenotung]

I think the browser only convert cases of the domain part, if the browser doesn't honor the whole url cases, ipfs address would have been broken already, but it's not broken.