Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cache server-side list of add-ons #4504

Open
Wedge009 opened this issue Oct 22, 2019 · 12 comments

Comments

@Wedge009
Copy link
Member

@Wedge009 Wedge009 commented Oct 22, 2019

Not terribly important, but I was thinking that caching the add-ons data would be a good idea. My Internet connection is pretty poor even by Australian standards and that, combined with the (presumably) slow connection from the add-ons server, means re-downloading (currently, as of 1.14.9 release) 725.4 KiB of data every time is very slow (several seconds). Of course, I suppose people are publishing all the time, so maybe a cache isn't a good idea (unless we can implement a system where new add-on data is published incrementally).

On the other hand, caching may help with #1904 and #3592. For the former, a cache would allow the system to remember which add-ons it's already seen. For the latter, an unresponsive server could allow the game to fall-back to a cache (if available) to at least browse what's available, even if an unresponsive server likely means that downloads won't be possible - or at best be very slow.

@soliton- soliton- changed the title Add-ons data cache Cache server-side list of add-ons Oct 23, 2019
@soliton-

This comment has been minimized.

Copy link
Member

@soliton- soliton- commented Oct 23, 2019

The server could increment a revision number for the addons list on every change. Then the client could send the revision it has cached when requesting the list and the server would only actually send the list if it has a newere revision.

Would have to figure out how to handle server restarts though.

@jostephd

This comment has been minimized.

Copy link
Member

@jostephd jostephd commented Oct 24, 2019

We could probably implement some custom revision number thing, but I feel like we're reinventing the wheel here. If we used rsync/git/svn, that would solve this issue, and #1458, and #1313, and probably a few others...

@AI0867

This comment has been minimized.

Copy link
Member

@AI0867 AI0867 commented Oct 24, 2019

I was about to comment that HTTP also already implements most of these things. I'm not entirely sure about automatic version control, given large binaries and add-on authors accidentally uploading sensitive files (that we then backup without their knowledge).

We also already have a webinterface for the add-on server.

@soliton-

This comment has been minimized.

Copy link
Member

@soliton- soliton- commented Oct 24, 2019

Using established protocols is certainly a good idea but changing the protocol goes well beyond this feature request.

And the question is where it ends. Converting campaignd to speak HTTP and directly integrating the webinterface sounds great. Then wesnoth already speaks HTTP so we might as well convert wesnothd to HTTP as well? Then what about the replay format, is that changed as well or just wrapped in HTTP? What about breaking compatibility with existing infrastructure?

I'm not at all suggesting this is a bad idea but it needs to be planned well and is a fair amount of work even for the smallest step of just converting campaignd to HTTP. There should probably be a new issue for this unless @Wedge009 is fine with repurposing this one.

@Wedge009

This comment has been minimized.

Copy link
Member Author

@Wedge009 Wedge009 commented Oct 24, 2019

The core of my thinking was that - for me, at least - opening the add-ons manager, closing it (whether accidental or deliberately - and then remembering to check for something else), and then re-opening it results in another download of the same information with the same relatively long wait time for the repeat opening. I thought one way to improve this experience is to cache the data and that caching could also help with the other issues I mentioned. If there are better ways to improve this situation I'm not against changing the request to match.

Regardless, this isn't something I do very often (open add-ons manager in quick succession). But I saw room for improvement and didn't see any existing issues relating to it, and so I made a note of it.

@jostephd

This comment has been minimized.

Copy link
Member

@jostephd jostephd commented Oct 25, 2019

And the question is where it ends. Converting campaignd to speak HTTP and directly integrating the webinterface sounds great. Then wesnoth already speaks HTTP so we might as well convert wesnothd to HTTP as well? Then what about the replay format, is that changed as well or just wrapped in HTTP? What about breaking compatibility with existing infrastructure?

@soliton- There is no reason why changes to downloading the list of addons would require changes to the replay format or to the wesnothd protocol, and no one has proposed to do that in any case. The proposal was simply to change the way the list of addons is downloaded; nothing more. Do you have any objections to that?

Directly integrating the addons web interface with the interface the in-game addon manager uses is an interesting idea, and I see how if we choose to pursue it we might postpone the fix to this issue until the integration is done, but again, that's just an if, not a certainty. We might choose to fix this issue first, and to do the integration later or not at all.

@soliton-

This comment has been minimized.

Copy link
Member

@soliton- soliton- commented Oct 25, 2019

As I tried to make quite clear I don't object. I was merely pointing out what it may mean to switch from WML to HTTP as a network protocol. Using HTTP only for the addon server sounds halfhearted (and only for the list of addons makes little sense at all so I assume that's not what was meant) since that seems to just increase complexity for the project as a whole. So if there is no bigger plan to generally switch to HTTP I'm not sure it's such a good idea.

Again this is not an objection just something to think about.

@Pentarctagon

This comment has been minimized.

Copy link
Member

@Pentarctagon Pentarctagon commented Oct 26, 2019

Is it possible to know how much of that 725.4 KiB are DataURIs (or I suppose exceptionally long IPF chains) vs, well, probably everything else?

If the icons are what's taking up the majority of the data, then a simpler, though somewhat less functional, option would be to add a preference to tell campaignd not send icons or perhaps to only send icons smaller than ~50-100 characters (since those would be just paths to hopefully mainline images).


On the other hand, using HTTP might make it easier to add encryption to the communication? IIRC all information is currently sent/received in the clear, including user emails when uploading an add-on as well as account passwords when logging into the MP server.

@jostephd

This comment has been minimized.

Copy link
Member

@jostephd jostephd commented Oct 26, 2019

as well as account passwords when logging into the MP server.

And since forum passwords and wesnothd passwords are the same...

@soliton-

This comment has been minimized.

Copy link
Member

@soliton- soliton- commented Oct 29, 2019

Authentication works with password hashes.

@soliton-

This comment has been minimized.

Copy link
Member

@soliton- soliton- commented Oct 29, 2019

Currently the icon= lines take up more than half the list data. Specifically the icon of Advanced Wesnoth Wars is 400+KB which is ten times the size of other icons.

Looks like we may need some limit there...

@Pentarctagon

This comment has been minimized.

Copy link
Member

@Pentarctagon Pentarctagon commented Oct 30, 2019

Though IIRC it's currently only using MD5, I think to continue supporting 1.12 clients?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.