Zite download incomplete when size > 20mb

[ghost]

When adding a zite that has more than 20MB, zeronet will ask only once if you want to increase the site size from 10MB to 20MB and it will leave the download incomplete with no indication of that fact.

To reproduce:

1. Open Antilibrary.bit -> It will ask you to increase the size from 10 to 20mb, accept that
2. Once the download seems to be done check the folder ZeroBundle\ZeroNet\data\147nKEb9bBJgBDgQytEQhmiGjF8e6HkEua\data\json, you should have 32 folders in there (al_1 - al_32) but you will have only 7
3. There are no indications that the site download is incomplete so the user won't notice it.

Expected behaviour:

1. ZeroNet should ask me if I want to increase the site size to +10mb from the total size calculated from the files in content.json.
2. Is should also give some indication of the download status (not the download progress bar) while in the site page. Something that will highlight to the user that he is missing files and the site download is not complete.

[ghost]

It seems that the problem has compounded as the zite (antilibrary.bit) got bigger.
The zite has now ~60mb and when I try do open it on Windows 10 zeronet rev 1529 I'll get:

[20:10:18] Site:147nKE..kEua content.json: Site too large 71768737 > 10485760, aborting task...

If I try to drag the zero icon in order to increase the allowed zite size I get:

[20:10:18] Site:147nKE..kEua Content.json not exist: data/147nKEb9bBJgBDgQytEQhmiGjF8e6HkEua/content.json
[20:10:19] Site:147nKE..kEua WebSocket handleRequest error: KeyError: 'content.json' in UiWebsocket.py line 99 > UiWebsocket.py line 176 > SidebarPlugin.py line 448 > SidebarPlugin.py line 130 > ContentManager.py line 271 > ContentDbDict.py line 54

So basically the zite cannot be downloaded.

[HelloZeroNet]

Thanks for reporting, i was able to reproduce and fix the problem, the modifications will be released soon!

[ghost]

@HelloZeroNet sorry mate, but, in which version has this been fixed?
I'm using Rev1536 and I still get:

[12:04:18] Site:147nKE..kEua Content.json not exist: data/147nKEb9bBJgBDgQytEQhmiGjF8e6HkEua/content.json
[12:04:19] Site:147nKE..kEua content.json: Site too large 167468609 > 10485760, aborting task...

[HelloZeroNet]

Yeah, it's not public yet. (I have made many modifications that needs more testing before relese)

It will be out with the next major version, that is planned for the next week.

[Osvalcosta]

is there any way to clone Antilibrary site i found it very interesting i want to clone it and use Hindu books

[HelloZeroNet]

Big site support fixed in 0.5.0

[ghost]

I'm testing the version 0.5.0 and the problem seems to persist.
If you try to download the site antilibrary.bit on a fresh zeronet install of 0.5.0 it will download only 10MB of an old version of the zite. Once the download is done nothing else happens. Even when I push a new version out, nothing happens on these nodes.
I'm not sure how ZeroNet pick the most up to date content.json file from the available peers but on my tests it seems that it will pick randomly, and if it picks from an up to date seed it will ask to increase the site size to 500MB but it will still download the old version that has 9984kB (and nothing else happens).
If it picks one of the outdated seeds it won't even ask to increase the site size limit, it will simply download the same old version.
The most up to date content.json file has "modified": 1478647623.986517.
The 10mb version of the zite has a 'Packs' item in the menu, the latest version has a 'My Books' item in the menu.
I won't update the content.json until you acknowledge the issue (so I won't modify the timestamp)
Thanks

ps: I've just noted that I'm getting {'to': 1, 'cmd': 'response', 'error': 'File invalid'} from peers holding the old version of the zite. This means they are not updating and are serving the old version when someone tries to reach the site for the first time.

[HelloZeroNet]

Yes, every peer is equal in zeronet, so when someone connects to the site it's try to download content.json from random peers, so if most of the peers has the old version then probably it will download that one.

[ghost]

@HelloZeroNet and what options does the site owner have to push changes that are not compatible with the old site version through the network? For example in these case some peers are stuck in the 10MB version and ZeroNet won't ask them to increase the size because it won't fetch the new content.json because the site has reached its size limit. How to force an update into these peers so they will at least be asked if they want to increase the site size?
thanks

[HelloZeroNet]

You can't force peers to increase size, so they will reject the invalid content.json files and serve the older ones until they increase the size.
So your options for larger sites:

• Merger sites: Create multiple merged sites eg based on categories or upload date (recommended)
• Add multiple content.json files using includes, so the root content.json will be always valid, but the included content.json files will be rejected if its larger than the allowed size
• Optional files: it's not for database/dynamic files, so it's not recommended

[ghost]

Don't you think that this can lead to a bad user experience?
The user has an outdated site but he is unaware that the site is outdated.
IMHO it would be a better UX if ZeroNet warns the user about that fact, something on the lines of: "- Hey, there's a new version of this site released on 11/11/2016 - 10:00PM. It has not been auto-updated because the new version is bigger then the site size limit you've set so the site update has been paused for now. If you want to update, increase the site size limit to at least XMB and manually update the site".

I'm reluctant to use Merger sites because, being books, I would need to separate the content by category and this would lead to too many categories. The UX to keep track and add them would be a pain.
Also, a decent book index doesn't use much space, even in the current state ~2 million books would not take more than a good quality movie (~3-4GB). That's because the site is holding only metadata for the books.
I hope that when ZeroNet implements the json compression, this size should be reduced by 80% during transfers, which is very acceptable.

I also believe that the people that will use this site is not 'HD space conscious', mostly are data hoarders that love to download huge packs of books from around the web. So spreading the site content into merger sites would affect their UX while at the same time not tackling a pain point for them.

Thoughts?
Thanks

[HelloZeroNet]

Yeah, a notification on ZeroHello is a good idea, i will check whats the possibilites on that.
You can store more than one category in one merger site, it does not requires much modification in the site code, it allows the users to choose which categories are they interested in and also allows anyone to create new data source or make modifications in the display logic and publish it as separate merger site.

[ghost]

Thanks for looking into this.

I'll check the merger site documentation. It makes sense to separate it by language and some broad category like fiction, non-fiction. Thanks for the advice.

[ghost]

@HelloZeroNet it seems that the original bug is affecting merger sites as well. I'm trying to add a big merger site but I'm getting Site:1KrGwP..mevq content.json: Site too large 333680764 > 10485760, aborting task...

To Reproduce:

• Go to Antilibrary.bit (hopefully you will get my seed at 185.21.216.182 - or else you will get an outdated version of the site)
• Click 'My Books' (top right) then 'Settings'
• Click 'Download' on the English language

This will try to add the merger site 1KrGwPRtnn77MsL35LBpNTEvvUhoLhmevq which is above 10MB.
All other merger sites work fine.
Thanks

[HelloZeroNet]

I have added a section for the sites that about to running out of limit:

[ghost]

@HelloZeroNet nice solution.
What about the merger site?
Currently when you try to add a merger site bigger than the limit it will fail silently with Site too large....
Maybe a solution can be to show a wrapperNotification asking the user to accept the size increase.

[antilibrary]

The merger site issue is now affecting Antilibrary's German books database because that DB is now bigger than the default allowed size.
Again, the problem happens when one tries to add a merger site which is bigger than the default allowed size. It will fail silently to the user and show the message Site too large... on the console.
A proper solution would be to inform the user about the size of the merger site and ask if he wants to proceed.

[haschimoto]

How does one increase limitation of MergerSite other than clicking on the MergerSite, which is usually "empty" frontend, then dragging the sidebar open and increase from there?

Since this is a bit confusing for new users, it would be great if there was an option directly on the startpage sidebar for the affected sites showing up in the "Running out of Space" section via the "..." dropdown menu, like "Increase SizeLimit" or "Adjust SizeLimit", which then opens the confirmation dialog to increase it.

[HelloZeroNet]

If a merged site is running out of limit and you click on it, then a dialog appears:

But yeah, this dialog could also appear if you browsing the merger site itself.

[Sispheor]

Sorry, maybe it's not the right place to ask the question but, is the 10 MB limit still current? I can't find anything about this in the main readme or in the doc.

[HelloZeroNet]

Yes, by default the sites are still limited to 10MB. (optional files included in that limit)

[sergei-bondarenko]

@HelloZeroNet Can this issue be closed?

[antilibrary]

I don't think so as the issue is still affecting merger sites.
If you try to add a merger site bigger than the default size it will fail silently. You do get the notification asking to add the merger site, but it doesn't add as the error happens in the background and there is no notification asking if the user wants to increase the merger site size.