ALL: Add Cloud storage support #788

merged 437 commits into from Aug 30, 2016


None yet

7 participants

Tkachov commented Jul 21, 2016

This adds the Cloud storages support feature. Four providers supported: Dropbox, OneDrive, Google Drive and Box. These are controlled by CloudManager, which could be accessed as CloudMan singleton. CloudMan saves the information in scummvm.ini within "cloud" section.

Storages use OAuth2 and interact with providers' API by giving the access_token. OneDrive, Google Drive and Box has to refresh token from time to time. To send the request, libcurl is used. There is a small system of Requests built on top of it. Requests are managed by ConnectionManager, which works in a separate thread (implemented with Timer) and keeps them running. If there are no more Requests, ConnectionManager stops the timer. While Storage is working (i.e. Requests are running), a pulsating Cloud icon is shown in the corner of the screen.

Cloud icon

Cloud-related information is available at the new Cloud tab of the Options dialog. One can switch connected storages there, refresh the information about these or connect a new one. The main features of the Cloud support are automatic saves sync (starts on each manual save, autosave, startup and on SaveLoadDialog opening) and games download (could be used from the Cloud tab). Before downloading ScummVM checks whether connection is "limited" and notifies user about that. Networking::Connection::isLimited() has only Android backend yet, where it checks whether Wi-Fi or a mobile network is used.

Options dialog Cloud tab

The Wi-Fi sharing feature is also added. ScummVM now can start a local webserver (using SDL_Net), which provides the Files Manager: users can navigate through directories, download files, create new directories or upload files using the browser on another device. Information about the server is available on the same Cloud tab. Users could change the webserver's port there.

AJAX Files Manager
LocalWebserver's port in Cloud tab

To connect a Storage, users must navigate to the link we show on the screen, allow our app access to their account, and receive the code which would be used by ScummVM to get the access_token. Some features were added in order to simplify that. First of all, we show a special short link. Secondly, there is a special "Open URL" button, which opens that link in a default browser, if the corresponding Networking::Browser::openUrl() backend is implemented.

Storage Connection Dialog: "Open URL" button, 8 fields

The code is quite long, so page transforms it into a few short groups with hashsum added. ScummVM dialog shows 8 fields, where user should type these groups, and checks that hashsum is OK. Then I also added SDL2 clipboard support in EditableWidget, so now users could paste these short groups into these fields. code page

Finally, if there is LocalWebserver available, a bit different link is shown and cloud providers redirect user to the localhost:12345 page instead of This way ScummVM can get the code without user typing it in the fields, which makes connecting a Storage extremely simple.

Storage Connection Dialog: no fields

It lacks iOS implementations of Networking::Browser::openUrl() and Networking::Connection::isLimited().

There are some commits from @sev- 's gui-improvement. This code was in my container-box branch while I was working on it, so there are clipping versions for transparent surfaces from gui-improvement. Yet, I had to remove it while preparing PR with Container box, so when I merged that back into my cloud branch, I didn't add those clipping versions back. In short, there are commits for transparent pictures, but those don't get clipped.

There is no code overriding loading Storage's KEY and SECRET from scummvm.ini.

All the short links are redirecting to giving access to my test applications. Actual ScummVM applications must be created with their real keys and secrets, and links must be edited to redirect to these.

Some TODOs are still in the code, I'm working on these.

@Tkachov Tkachov changed the title from Add Cloud storages support to ALL: Add Cloud storages support Jul 21, 2016
@Tkachov Tkachov changed the title from ALL: Add Cloud storages support to ALL: Add Cloud storage support Jul 21, 2016
@uruk uruk changed the title from ALL: Add Cloud storage support to CLOUD: Add cloud storages support Jul 21, 2016
@Tkachov Tkachov changed the title from CLOUD: Add cloud storages support to ALL: Add Cloud storage support Jul 21, 2016
@wjp wjp commented on an outdated diff Jul 21, 2016
@@ -784,6 +785,8 @@ SaveStateList SciMetaEngine::listSaves(const char *target) const {
return saveList;
+bool SciMetaEngine::simpleSaveNames() const { return true; }
wjp Jul 21, 2016 Member

SCI usually uses standard savegame names, but not for all games, and some games additionally store other files using the savegame system (containing settings and such).

wjp Jul 21, 2016 Member

To elaborate, the SCI engine could give a list of all savegames (and other saved files) for the current game, but it won't be limited to the usual target.###.

@wjp wjp and 1 other commented on an outdated diff Jul 21, 2016
@@ -116,6 +116,20 @@ class MetaEngine : public PluginObject {
+ * Return whether engine's saves could be detected with
+ * "<target>.###" pattern and "###" corresponds to slot
+ * number.
+ *
+ * If that's not true or engine is using some unusual way
+ * of detecting saves and slot numbers, this should return
+ * false. In that case Save/Load dialog would be unavailable
+ * during cloud saves sync.
+ *
+ * @return true, if "<target>.###" is OK for this engine
+ */
+ virtual bool simpleSaveNames() const { return false; }
wjp Jul 21, 2016 Member

Wouldn't the simpleSaveNames thing fit better as an engine feature? (See MetaEngineFeature)

Tkachov Jul 21, 2016 Contributor

Well, it looks like it really could be implemented as an engine feature. Didn't see that back then. Noted.

@wjp wjp commented on the diff Jul 21, 2016
+ _stopOnIdle = false;
+ if (_timerStarted) {
+ _handleMutex.unlock();
+ return;
+ }
+ startTimer();
+ // Create a listening TCP socket
+ IPaddress ip;
+ if (SDLNet_ResolveHost(&ip, NULL, _serverPort) == -1) {
+ error("SDLNet_ResolveHost: %s\n", SDLNet_GetError());
+ }
+ resolveAddress(&ip);
+ _serverSocket = SDLNet_TCP_Open(&ip);
wjp Jul 21, 2016 Member

I'm not too familiar with SDLNet specifically, but why not only listen on

Tkachov Jul 21, 2016 Contributor

Well, passing NULL up there means we want to use ("listen on every available network interface").

Address resolving is needed because local webserver (in case of Wi-Fi sharing feature) is meant to be accessed from another device on the same network, thus actual local IP is required instead of plain

wjp Jul 21, 2016 Member

What is a wifi sharing feature, why do we need it, and where I can find information about your use of it?

sev- Jul 21, 2016 Member

This is a local webserver running on top of ScummVM, so a user can open the provided URL in their local network and have direct access to the device filesystem. Then they can upload the game files or download saves from their device running ScummVM.

This is an alternative to the external cloud usage.

Tkachov Jul 21, 2016 Contributor

Some information about it is in the PR description and in my blog.

We need it because it's cool and some other apps - for example, VLC - have such feature.

Almost all "handlers" in the backends/networking/sdl_net/ are implementing this feature.

In short, it allows one to access files via network - upload and download files from the device ScummVM is working on. For example, one can upload a game from a PC onto a smartphone without USB cable.

wjp Jul 21, 2016 Member

Hm, I see how that could be nice. Is the same local webserver used for this feature and the OAuth stuff?

Tkachov Jul 21, 2016 Contributor

Yes, it is.

wjp Jul 21, 2016 Member

That has the unfortunate side effect of opening a large attack surface unnecessarily when not using this feature.

Tkachov Jul 21, 2016 Contributor

Webserver has to be turned on manually with a special button in the Cloud tab.

It's also turned on automatically while connecting a Storage, but it's turned off when Storage is connected.

As ScummVM isn't working on routers, I doubt someone could access files with that feature through the Internet. If the user is smart enough to configure port forwarding, I guess he/she is smart enough to use that with caution.

If the malefactor is on the same local network... well, user shouldn't have run the server while using a non-trusted network, then.

wjp Jul 21, 2016 Member

Is that really your attitude towards security?

sev- Jul 21, 2016 Member

We need to ensure that there is no way to go beyond a sandboxed directory where all game-related files could be stored. Also, ScummVM is not supposed to execute anything directly, all code in the engines so far is using VMs.

My recommendation is to enforce the server close once the dialog with upload files is closed. In this case this will be yet another safety feature.

However, we are using the local server for accepting keys from the local browser (e.g. we provide as a callback URL). In this case I would enforce additional checks on the incoming URL formats. We are using it only for extracting the keys provided in the GET URLs

Tkachov Jul 21, 2016 Contributor

As I mentioned, server is turned off automatically after it's used in the Storage connection. As Eugene said, we can additionally turn off the server when user closes the Options dialog, so it would be available only while user actually works with it.

wjp commented Jul 21, 2016

Given the big impact of this, is there any technical design document? (In particular about the various security considerations.)

sev- commented Jul 21, 2016

Yes, it was highlighted in the Alex's blog. For instance, see here:

@sev- sev- commented on the diff Jul 21, 2016
@@ -39,6 +39,20 @@ void BaseBackend::displayMessageOnOSD(const char *msg) {
+void BaseBackend::copyRectToOSD(const void *buf, int pitch, int x, int y, int w, int h) {
+ warning("BaseBackend::copyRectToOSD not implemented"); //TODO
+void BaseBackend::clearOSD() {
+ warning("BaseBackend::clearOSD not implemented"); //TODO
+ //what should I do? Remove all TimedMessageDialogs?
sev- Jul 21, 2016 Member

Yes, skip those. Additionally, we could add another backend feature: kFeatureOSD.

@sev- sev- and 1 other commented on an outdated diff Jul 21, 2016
+ //debug("%s", json->stringify(true).c_str());
+ //TODO: check that error is returned the right way
+ /*
+ if (responseObject.contains("error") || responseObject.contains("error_summary")) {
+ warning("Box returned error: %s", responseObject.getVal("error_summary")->asString().c_str());
+ error.failed = true;
+ error.response = json->stringify();
+ finishError(error);
+ delete json;
+ return;
+ }
+ */
+ //TODO: check that ALL keys exist AND HAVE RIGHT TYPE to avoid segfaults
sev- Jul 21, 2016 Member

Right, it is a must for implementing. Could also help with testing if the providers change their API.

Tkachov Jul 21, 2016 Contributor

Yep, that's what I'm working on.

@sev- sev- commented on an outdated diff Jul 21, 2016
+ if (responseObject.contains("entries") && responseObject.getVal("entries")->isArray()) {
+ Common::JSONArray items = responseObject.getVal("entries")->asArray();
+ for (uint32 i = 0; i < items.size(); ++i) {
+ Common::JSONObject item = items[i]->asObject();
+ Common::String id = item.getVal("id")->asString();
+ Common::String name = item.getVal("name")->asString();
+ bool isDirectory = (item.getVal("type")->asString() == "folder");
+ uint32 size = 0, timestamp = 0;
+ if (item.contains("size")) {
+ if (item.getVal("size")->isString())
+ size = item.getVal("size")->asString().asUint64();
+ else if (item.getVal("size")->isIntegerNumber())
+ size = item.getVal("size")->asIntegerNumber();
+ else
+ warning("strange type for field 'size'");
sev- Jul 21, 2016 Member

It is always helpful to print out the unexpected value when possible.

@sev- sev- and 1 other commented on an outdated diff Jul 21, 2016
+ }
+ uint32 received = 0;
+ uint32 totalCount = 0;
+ if (responseObject.contains("total_count") && responseObject.getVal("total_count")->isIntegerNumber())
+ totalCount = responseObject.getVal("total_count")->asIntegerNumber();
+ if (responseObject.contains("offset") && responseObject.getVal("offset")->isIntegerNumber())
+ received = responseObject.getVal("offset")->asIntegerNumber();
+ if (responseObject.contains("limit") && responseObject.getVal("limit")->isIntegerNumber())
+ received += responseObject.getVal("limit")->asIntegerNumber();
+ bool hasMore = (received < totalCount);
+ if (hasMore) makeRequest(received);
+ else finishListing(_files);
+ } else {
+ warning("null, not json");
sev- Jul 21, 2016 Member

Please put prefix, like: "BoxListDirectoryByIdRequest::responseCallback(): null, not json". Will help greatly with the troubleshooting.

Tkachov Jul 21, 2016 Contributor


bluegr commented Jul 21, 2016

From a quick glance: quite impressive work, overall! Well done! :)

I'll check this out later on

wjp commented Jul 21, 2016

I mean a design document for the whole general cloud functionality. This feature will require long-term maintainability more than most of our current features, and also potentially urgent security fixes
sometimes. I don't think blog posts or a pull request description (while nice) are really sufficient for this.

@sev- sev- commented on the diff Jul 21, 2016
+ }
+ Networking::JsonCallback innerCallback = new Common::CallbackBridge<BoxStorage, BoolResponse, Networking::JsonResponse>(this, &BoxStorage::tokenRefreshed, callback);
+ if (errorCallback == nullptr) errorCallback = getErrorPrintingCallback();
+ Networking::CurlJsonRequest *request = new Networking::CurlJsonRequest(innerCallback, errorCallback, "");
+ if (codeFlow) {
+ request->addPostField("grant_type=authorization_code");
+ request->addPostField("code=" + code);
+ } else {
+ request->addPostField("grant_type=refresh_token");
+ request->addPostField("refresh_token=" + _refreshToken);
+ }
+ request->addPostField("client_id=" + Common::String(KEY));
+ request->addPostField("client_secret=" + Common::String(SECRET));
+ /*
+ if (Cloud::CloudManager::couldUseLocalServer()) {
sev- Jul 21, 2016 Member

Is this working?

Tkachov Jul 26, 2016 edited Contributor

Short links work fine. And it looks like Box doesn't check the redirect_uri parameter when token is requested.

But I just checked Box docs/app editing page... Box allows only one redirect_uri. And localhost option is available for apps in development only. I guess we'd have to use option for Box all the time, even if LocalWebserver is available.

@sev- sev- commented on an outdated diff Jul 21, 2016
+ }
+ */
+ addRequest(request);
+void BoxStorage::tokenRefreshed(BoolCallback callback, Networking::JsonResponse response) {
+ Common::JSONValue *json = response.value;
+ if (!json) {
+ warning("BoxStorage: got NULL instead of JSON");
+ if (callback) (*callback)(BoolResponse(nullptr, false));
+ return;
+ }
+ Common::JSONObject result = json->asObject();
+ if (!result.contains("access_token") || !result.contains("refresh_token")) {
+ warning("Bad response, no token passed");
sev- Jul 21, 2016 Member

Same thing, this need to have a prefix, so you could find it in the source code with ease.

@sev- sev- commented on an outdated diff Jul 21, 2016
+ */
+ addRequest(request);
+void BoxStorage::tokenRefreshed(BoolCallback callback, Networking::JsonResponse response) {
+ Common::JSONValue *json = response.value;
+ if (!json) {
+ warning("BoxStorage: got NULL instead of JSON");
+ if (callback) (*callback)(BoolResponse(nullptr, false));
+ return;
+ }
+ Common::JSONObject result = json->asObject();
+ if (!result.contains("access_token") || !result.contains("refresh_token")) {
+ warning("Bad response, no token passed");
+ debug("%s", json->stringify().c_str());
sev- Jul 21, 2016 Member

Please hide this to some deep level, like 7 or deeper.

@sev- sev- commented on an outdated diff Jul 21, 2016
+ return;
+ }
+ if (outerCallback) {
+ Common::JSONObject info = json->asObject();
+ (*outerCallback)(BoolResponse(nullptr, info.contains("id")));
+ delete outerCallback;
+ }
+ delete json;
+Networking::Request *BoxStorage::createDirectoryWithParentId(Common::String parentId, Common::String name, BoolCallback callback, Networking::ErrorCallback errorCallback) {
+ if (!errorCallback) errorCallback = getErrorPrintingCallback();
+ Common::String url = "";
sev- Jul 21, 2016 Member

I recommend to move these to the very top of the file as constants with #define.

Tkachov commented Jul 21, 2016

Sure, I'd compile the detailed information on a wiki page.

@sev- sev- commented on an outdated diff Jul 21, 2016
+ Networking::ErrorResponse error(this, false, true, "", -1);
+ Networking::CurlJsonRequest *rq = (Networking::CurlJsonRequest *)response.request;
+ if (rq) {
+ const Networking::NetworkReadStream *stream = rq->getNetworkReadStream();
+ if (stream) {
+ long code = stream->httpResponseCode();
+ error.httpResponseCode = code;
+ }
+ }
+ if (error.httpResponseCode != 200 && error.httpResponseCode != 201)
+ warning("looks like an error");
+ Common::JSONValue *json = response.value;
+ if (json) {
sev- Jul 21, 2016 Member

For simplicity I would negate it. E.g.:

if (!json) {
   warning("BoxUploadRequest::uploadedCallback(): null, not json");

if (!json->isObject()) {

Common::JSONObject object = json->asObject();
sev- commented Jul 22, 2016

I just rebased everything to the tip of the master

Tkachov commented Jul 22, 2016

I've added a detailed description of Cloud and Local Webserver / Wi-Fi Sharing features to the wiki:
Cloud Storage Support
Local Webserver

uruk commented Jul 22, 2016

I also added the two documentation pages to our Developer Central:

@wjp wjp commented on the diff Jul 22, 2016
+ continue;
+ }
+ break;
+ if (_headersStream == nullptr)
+ _headersStream = new Common::MemoryReadWriteStream(DisposeAfterUse::YES);
+ if (client->readBlockHeaders(_headersStream)) {
+ handleBlockHeaders(client);
+ continue;
+ }
+ break;
+ // _contentStream is created by handleBlockHeaders() if needed
wjp Jul 22, 2016 Member

I don't see how you're handling the error case where it isn't created.

Tkachov Jul 26, 2016 Contributor

Woah, I thought the comment is about new up there and handling a situation when object is not allocated. Well, I'm not handling such situations mostly, because it's difficult to do something when you can't allocate more memory.

As for _contentStream, if it isn't created, it's nullptr. It's passed to Client::readBlockContent(). As mentioned in the wiki (and also in Client's description comment), Client uses Reader internally.

And when you pass nullptr as a stream to Reader, it reads contents into the void (reads, sees there is no stream, does nothing). So, passing nullptr here is similar to > /dev/null in a terminal.

criezy commented Jul 24, 2016

I have not looked at the code yet, but from the description in this PR I have one suggestion and one question.

The suggestion would be to have a "Paste from Clipboard" button in the tab where we need to enter the access token. From what I understand, the way pasting is implemented currently we would have to do it eight time for each pieces fo the access token? The possibility to paste the full token in one go would be great. Also I am assuming currently pasting is done using a keyboard shortcut? In that case the button would help on devices that do not have a keyboard (it suspect it would at least be easier to implement than the long press usually used to popup the copy/paste menu, although that would be great too :P).

The question is about the local server and how tied it is to SDL Net? Is it implemented in such a way (e.g. with abstraction layers) that it would be easy for porters for platforms that do not use SDL to implement it in a different way?

criezy commented Jul 24, 2016

There are various warnings and compilation errors.

Some examples of warnings:

engines/testbed/cloud.cpp:121:82: warning: format specifies type 'unsigned int' but the argument has type 'const char *' [-Wformat]
                Testsuite::logPrintf("Info! First directory is '%u' and first file is '%s'\n", directory.c_str(), file.c_str());
                                                                ~~                             ^~~~~~~~~~~~~~~~~
engines/testbed/cloud.cpp:137:95: warning: format specifies type 'unsigned long' but the argument has type 'uint32' (aka 'unsigned int')
        Testsuite::logPrintf("Info! It's id = '%s' and size = '%lu'\n",, response.value.size());
                                                               ~~~                                   ^~~~~~~~~~~~~~~~~~~~~
engines/testbed/cloud.cpp:154:95: warning: format specifies type 'unsigned long' but the argument has type 'size_type' (aka 'unsigned int')
                Testsuite::logPrintf("Warning! %lu files were not downloaded during folder downloading!\n", response.value.size());
                                               ~~~                                                          ^~~~~~~~~~~~~~~~~~~~~
./gui/storagewizarddialog.h:49:7: warning: private field '_stopServerOnClose' is not used [-Wunused-private-field]
        bool _stopServerOnClose;

And some example of errors (disclaimer: I have disabled the use of SDL Net):

gui/options.cpp:1322:28: error: no member named 'LocalWebserver' in namespace 'Networking'
        uint32 port = Networking::LocalWebserver::getPort();
gui/storagewizarddialog.cpp:198:7: error: use of undeclared identifier 'kStorageCodePassedCmd'
        case kStorageCodePassedCmd:
gui/storagewizarddialog.cpp:199:39: error: use of undeclared identifier 'LocalServer'
                CloudMan.connectStorage(_storageId, LocalServer.indexPageHandler().code());

I am aware this is all code in progress, so I will stop there in error reporting. But they should all be fixed before this PR is merged.

Tkachov commented Jul 25, 2016

Good suggestion about the "Paste from clipboard" button! Right now it works when built with SDL2, via Ctrl+V shortcut.

LocalWebserver class itself uses SDL_Net sockets (and socket set). Client does too. Other classes interact with these, so these would use sockets and return some result. SDL_Net sockets are not really complex, and I believe could be easily replaced with usual sockets (IIRC there were socket sets of some kind in POSIX). Yet, in order to use other sockets instead of SDL_Net's, we'd have to introduce a few own classes, as I didn't.

I fixed compilation without SDL_Net or libcurl a few times, but I guess I've broken that again. One of the warnings you've posted is really useful (%u for string somehow). I've used a lot of uint64, so I guess I used %lu automatically in the other two. We could avoid the last one, if we'd hide the field with #ifdef, but after that we might end up having an undefined identifier error instead. I'll try to fix these.

@Tkachov Tkachov added a commit to Tkachov/scummvm that referenced this pull request Jul 25, 2016
@Tkachov Tkachov ALL: Make simpleSaveNames() a MetaEngineFeature
Added it into hasFeature() of all engines which returned `true` in
simpleSaveNames() before.

As mentioned in #788, SCI is not always using simple names, so it
doesn't have such feature now.
criezy commented Jul 25, 2016

I had some time to look a bit more at the code. Great work!
However I have one more warning and a couple of remarks/suggestions you might want to discuss with your mentor.

First the warning is:

gui/storagewizarddialog.cpp:50:43: warning: commas at the end of enumerator lists are a C++11 extension [-Wc++11-extensions]
        kStorageWizardContainerReflowCmd = 'SWCr',

The first remark is about the copy/paste feature. I am thinking that rather than use directly SDL2 in the gui code it might be better to extend OSystem and let backends implement the functionality (with for example SDL2, but not necessarily). For example you might want in OSystem to add:

  • kFeatureClipboard
  • bool hasTextInClipboard
  • String getTextFromClipboard

(and if we need copy at some point we can add a copyTextToClipboard)

The second remark is about the use of CURLOPT_XFERINFOFUNCTION, which was added in curl 7.32 (which is "only" 3 years old :P ). This causes compilation to fails on my system. I suppose I could compile a more recent curl manually and use that, but I am wondering about the possibility to use CURLOPT_PROGRESSFUNCTION rather than CURLOPT_XFERINFOFUNCTION to support a wider range of curl versions and ease the pain for port maintainers.

Tkachov commented Jul 26, 2016

Got it about the clipboard, would move it to OSystem.

While reading the docs, I thought CURLOPT_PROGRESSFUNCTION is deprecated. It actually is "obsolete". In their code they use both by adding #if LIBCURL_VERSION_NUM >= 0x072000. I'll do the same then.

@mgerhardy mgerhardy commented on an outdated diff Jul 28, 2016
@@ -64,6 +64,11 @@
#include <SDL/SDL_net.h>
mgerhardy Jul 28, 2016 Contributor

This looks weird - the SDL_VERSION_ATLEAST macro should only be known if you already included SDL.h.

also (not sure about the scummvm guidelines) it looks wrong to include SDL2/SDL.h - just include SDL.h and let the include paths be configured correctly (e.g. pkg-config --cflags sdl2 will print /usr/include/SDL2 on my system - and this is also the default that SDL2 configures in their pc files). If you already include SDL.h for version >= 2.0.0, SDL_clipboard.h is also automatically included.


I've read in your blog post that ios openurl is still missing, check out this:

Tkachov commented Jul 29, 2016

@mgerhardy, yeah, I guess that should do. I can add it then, but I have no means to check that it compiles and works fine.


@Tkachov You know that Travis CI also supports macosx and thus ios builds, no? See e.g here: - quite useful, as I also don't fire up my mac for each commit and don't even have windows (see appveyor ci - e.g.

Tkachov commented Jul 31, 2016

@mgerhardy Well, I like checking that something compiles and works before I commit it, not after. Plus, I'm not really sure how these .mm work, or which headers are required (in that example you've posted <SDL.h> is included though I don't think it's needed), or whether I'd need to define FORBIDDEN_SYMBOL_ALLOW_ALL, etc.

I mean, I can try commiting something I think should compile and work, then look at the result, try fixing something and then commiting again until it works... But I'd rather leave iOS code to someone who knows how to handle it.

@Tkachov Tkachov added a commit to Tkachov/scummvm that referenced this pull request Aug 3, 2016
@Tkachov Tkachov ALL: Make simpleSaveNames() a MetaEngineFeature
Added it into hasFeature() of all engines which returned `true` in
simpleSaveNames() before.

As mentioned in #788, SCI is not always using simple names, so it
doesn't have such feature now.
Tkachov and others added some commits Jun 15, 2016
@Tkachov Tkachov CLOUD: Add comments for StorageWizardDialog methods f571f3d
@Tkachov Tkachov CONFIGURE: Add --with-sdlnet-prefix option e2b3a93
@Tkachov Tkachov CLOUD: Init SDL_Net 9f7bea1
@Tkachov Tkachov CLOUD: Add LocalWebserver
Available as LocalServer singleton. It's being started and stopped by
StorageWizardDialog. It doesn't handle clients yet, though.
@Tkachov Tkachov CLOUD: Add Networking::Client
Keeps current client's state
@Tkachov Tkachov CLOUD: Add ClientState::BAD_REQUEST 99c5138
@Tkachov Tkachov CLOUD: Add GetClientHandler
That ClientHandler is made for responding GET requests. It calculates
stream's length, it allows to specify response code and headers, it can
be used to transfer any ReadStream.
@uruk @Tkachov uruk CLOUD: Remove a couple of unnecessary whitespaces 434b740
@Tkachov Tkachov CLOUD: Minor Client fix 6e1b667
@Tkachov Tkachov CLOUD: Add HTTP response codes in GetClientHandler 892c1bf
@Tkachov Tkachov CLOUD: Clarify calculatedChecksum's initial value ceb86a0
@Tkachov Tkachov CLOUD: Prepare code for path handlers
LocalWebserver would storage the handlers.

Client now has methods like path() or query() to access different parts
of the request.
@Tkachov Tkachov CLOUD: Minor Client fix 733d998
@Tkachov Tkachov CLOUD: Add IndexPageHandler
This commit also adds LocalWebserver's stopOnIdle().
That means server is not stopped immediately, but only when all clients
are served.
@Tkachov Tkachov CLOUD: Fix Client
Cleanup in open()
@Tkachov Tkachov CLOUD: Update LocalWebserver
* fix handling connections;
* fix idling strategy;
* add setClientGetHandler() for SeekableReadStream;
* add determineMimeType().
@Tkachov Tkachov CLOUD: Add wwwroot contains ScummVM local webserver's resources, such as
template html pages, styles and images.

One can make it from wwwroot directory contents by running script.

It's added to scummvm.rc, so it's included in the executable (it works
with MinGW, but I was unable to do that in VS yet).

IndexPageHandler is the one who returns these resources. It uses
index.html for "/". I'm replacing "{message}" with translated message,
so that's the way I thought the templates should work.
@Tkachov Tkachov GUI: Hide StorageWizardDialog fields if server present c2c2ba9
@Tkachov Tkachov CLOUD: Update IndexPageHandler to search
Now it also searches for that in themepath, not with SearchMan only.
@Tkachov Tkachov CLOUD: Embed cloud icons as byte arrays 5e70f64
@Tkachov Tkachov CLOUD: Add some mutexes in LocalWebserver 6ac6972
@Tkachov Tkachov CLOUD: Use correct redirect_uris
Usage of #ifdef there (and in StorageWizardDialog) means that ScummVM
doesn't support both local webserver and paths at the same
time. It's either built with SDL_net (thus supporting localhost path) or
without it (thus using
@Tkachov Tkachov CLOUD: Fix "signed/unsigned integers" warning
The "comparison between signed and unsigned integer expressions" one.

Note that in UploadRequests size() and pos() are acutally signed,
because they could return -1. This commit implies that Requests are
working with such Streams which doesn't.
@Tkachov Tkachov CLOUD: Fix "type qualifiers ignored" warning 04cef89
@Tkachov Tkachov CLOUD: Fix "zero-length format string" warning 39eb76f
@Tkachov Tkachov CLOUD: Fix "-Wconversion-null"
That `false` came from TranslationManager's function, which was
returning bool, not a pointer. Somehow missed that line.
@Tkachov Tkachov CLOUD: Fix "-Wcast-qual"
The passed buffer is not changed, so could be `const`.

You might see that `postFields.c_str()` is `buffer`. Yet, as it's
`postFields`, it's used for POST in curl_easy_setopt(), which copies the
passed buffer. When `buffer` is used for upload, it's an actual bytes
buffer, kept in CurlRequest.
@Tkachov Tkachov CLOUD: Update local server's style.css 81106b0
@Tkachov Tkachov CLOUD: Make OutSaveFile start saves sync
It had to become a proxy class in order to do that.
finalize() starts the saves sync.
@Tkachov Tkachov CLOUD: Update save's timestamp on rewrite
This commit moves save/load timestamps static methods into
DefaultSaveFileManager and fixes a few related bugs.
@Tkachov Tkachov KYRA: Fix openSaveForWriting() to return OutSaveFile 2d3cfff
@Tkachov Tkachov CLOUD: Delete the incomplete file (when downloading) 4c381da
@uruk @Tkachov uruk CLOUD: Fix crash on exiting ScummVM while ConnMan is active cff1835
@Tkachov Tkachov CLOUD: Fix some warnings
Mostly on format string
@Tkachov Tkachov CLOUD: Fix finishSuccess() warning f3a3923
@Tkachov Tkachov CLOUD: Fix saves sync
Tested that on actual unix system and found out a few minor bugs related
to paths.
@Tkachov Tkachov GUI: Add DownloadDialog sketch 97c0bbd
@Tkachov Tkachov GUI: Add RemoteBrowserDialog
WIP. Tested with Dropbox.
@Tkachov Tkachov GUI: Fix "Go up"
OneDrive and Google Drive paths do not start with '/', so one was unable
to go up to root.
@Tkachov Tkachov GUI: Make RemoteBrowser show "Loading..." 4aa8e23
@Tkachov Tkachov GUI: Clean up in RemoteBrowser 73bb2e2
@Tkachov Tkachov GUI: Fix RemoteBrowser Request handling
Init with NULL, ignore callbacks, and such.
@Tkachov Tkachov GUI: Add RemoteBrowser parent directories remembering
No wait when "Go up" is pressed. These contents could be invalid,
though. In order to refresh contents, one has to go up one more time and
then get back inside (in root folder - just press "Go up" to refresh
@Tkachov Tkachov CLOUD: Make Google Drive sort files list
GoogleDriveListDirectoryByIdRequest now uses "orderBy" field to specify
that we want the commonly used "alphabetical, folders first" order.

That's mostly needed for RemoteBrowserDialog, because Requests don't
care about the order, and this one is more user-friendly.
@Tkachov Tkachov GUI: Add RemoteBrowser file list sorting
Because Dropbox has no means to specify files order.
@Tkachov Tkachov GUI: Add error message in RemoteBrowser
For the error callback case.
@Tkachov Tkachov GUI: Use RemoteBrowser's result in DownloadDialog
It now checks the selected local directory, and shows different
MessageDialogs to notify user of mistake or ambiguous situation.
@Tkachov Tkachov CLOUD: Add CloudManager::downloadFolder() dc0a956
@Tkachov Tkachov CLOUD: Fix CloudManager's methods
Were not returning created Request.
@Tkachov Tkachov GUI: Initiate download in DownloadDialog 71a3264
@Tkachov Tkachov CLOUD: Add FolderDownload-related methods in Storage
CloudManager's shortcuts are added too.

The idea is to keep FolderDownload request within Storage, and provide
necessary means to access it. The download is started and cancelled
through the DownloadDialog.
@Tkachov Tkachov GUI: Upgrade DownloadDialog
It now shows the remote and local directories and a progress bar.

Storage now shows OSD messages on download success and failure.
@Tkachov Tkachov GUI: Fix DownloadDialog path creation
Was adding a path separator even when none is required.
@Tkachov Tkachov GUI: Forbid using download directory in "Add Game" 052d8bf
@Tkachov Tkachov GUI: Fix SaveLoadDialog
It was SavesSyncRequest's target even when closed.
@Tkachov Tkachov CLOUD: Fix FolderDownloadRequest
Actually, I'm not completely sure, but this fixed the segfault when user
closes ScummVM during the download. Even if that's not a fix, these
lines must be in this method anyway.
@Tkachov Tkachov GUI: Update DownloadDialog
It now less empty, because if there is no download in progress, user
sees the RemoteBrowser instead of empty dialog. The cancel button is now
in the left bottom corner.
@Tkachov Tkachov GUI: Add lowres support for DownloadDialog's button
No actual translations, though. Should be just "Cancel", because "Cancel
download" is too long for lowres mode.
@Tkachov Tkachov GUI: Add "Run server" button in Cloud tab ad069f4
@Tkachov Tkachov CLOUD: Fix SaveLoadDialogs to check USE_CLOUD
Linking was failing when disabling curl support.
@Tkachov Tkachov GUI: Fix Options' Cloud tab reflowing 211d9ed
@Tkachov Tkachov CLOUD: Replace USE_CLOUD with USE_LIBCURL
In most cases that's the right one to check. USE_CLOUD is defined when
either USE_LIBCURL or USE_SDL_NET are, which means if there is no curl,
USE_CLOUD still could be defined and linking errors would appear.
@Tkachov Tkachov CLOUD: Make "Run server" button active
It should show the real server's IP over there, but that doesn't work
Tkachov and others added some commits Jul 18, 2016
@Tkachov Tkachov GUI: Add Cloud-related dialogs in classic theme's stx
Looks fine.
@Tkachov Tkachov CLOUD: Add ListAjaxHandler
"/list" now returns JSON with directory information. Would be used in
AJAX-based Files Manager.
@Tkachov Tkachov CLOUD: Add "/filesAJAX" sketch
It works already, but still requires some polishing.
@Tkachov Tkachov CLOUD: Add "breadcrumbs" in "/filesAJAX" 6442dad
@Tkachov Tkachov CLOUD: Minor "/filesAJAX" fix cd6d45e
@Tkachov Tkachov CLOUD: Add messages in "/filesAJAX" e6caa48
@Tkachov Tkachov CLOUD: Add "ajax" parameter for "/create" and "/upload"
If it's set, these redirect to "/filesAJAX" instead of "/files".
@Tkachov Tkachov CLOUD: Make "/create" support AJAX
Now creating directories doesn't refresh the "/filesAJAX" page.
@Tkachov Tkachov CLOUD: Move Dropbox to API v2
We had a few places where their deprecated API v1 was used.
@Tkachov Tkachov CLOUD: Fix DropboxCreateDirectoryRequest
It now calls success callback with `false` on Dropbox's
"path/conflict/folder", indicating that the directory already exists.
@Tkachov Tkachov CLOUD: Fix OneDriveUploadRequest
OneDrive doesn't accept empty files, so UploadRequest just skips such.
@Tkachov Tkachov CLOUD: Add Networking::Connection::isLimited()
`false` everywhere by default, but works on Android (`true` if not
@uruk @Tkachov uruk CLOUD: Fix makefile 8e9d106
@Tkachov Tkachov GUI: Show warning in DownloadDialog
If user's connection seems limited, ScummVM shows a warning message to
prevent using that by accident.
@Tkachov Tkachov CLOUD: Add default SavesSync callbacks
With OSD messages indicating whether saves sync is complete, cancelled
or failed.
@Tkachov Tkachov CLOUD: Fix OneDriveUploadRequest
Segfault when given stream is nullptr.
@Tkachov Tkachov CLOUD: Fix Dropbox and Google Drive UploadRequests
Possible segfault there too.
@Tkachov Tkachov GUI: Add error callback in Options' Cloud tab
Shows OSD message.
@Tkachov Tkachov CLOUD: Add OSD warning when can't start LocalWebserver e833c8f
@Tkachov Tkachov CLOUD: Fix backends/
MinGW failed to compile with the latest fix.

Checked this fix with create_project for MSVC, MinGW's make, make under
kubuntu and while building Android apk.
@Tkachov Tkachov CLOUD: Fix CloudManager::connectStorage() memory leak f743b31
@Tkachov Tkachov CLOUD: Check whether Storage is working when replacing it
We do that in CloudManager::replaceStorage(), but I've tried to
eliminate such possibility by adding a check in the StorageWizardDialog.
@Tkachov Tkachov CLOUD: Add port override for LocalWebserver

It's not defined, because override means we have to reconfigure our
redirect links somehow to use the override port.
@Tkachov Tkachov CLOUD: Update StorageWizardDialog
It now hides code fields not just when built with SDL_Net, but also when
LocalWebserver's using default port.

in localwebserver.h now.
@Tkachov Tkachov GUI: Add "Clear port" button in Cloud tab 52503a2
@Tkachov Tkachov GUI: Fix texts clipping
If it was completely clipped out (empty rectangle), it was drawing the
whole text ("empty means no clipping"), so I had to detect such cases
and change textArea to one small pixel.
@Tkachov Tkachov CLOUD: Fix Cppcheck warnings a449ddc
@Tkachov Tkachov CLOUD: Do some refactoring/cleanup
Nothing really major.
@Tkachov Tkachov CLOUD: Do some refactoring/cleanup in Networking 01161ae
@uruk @Tkachov uruk CLOUD: Fix code formatting 9254df2
@uruk @Tkachov uruk CLOUD: Remove remove() from BoxStorage 8c62993
@Tkachov Tkachov JANITORIAL: Remove spaces at the end of the line
I knew there were some, but I wanted to fix them once, instead of doing
it all the time.
@Tkachov Tkachov CLOUD: Fix `redirect_uri` selection code
Now it's not hardcoded based on USE_SDL_NET, but one or another value is
used depending on currently selected LocalWebserver's port.
@Tkachov Tkachov CLOUD: Remove DropboxStorage::remove() efebb5b
@Tkachov Tkachov CLOUD: Fix some TODOs in CloudManager
"No Storage connected!" error message is passed to the error callback
now when there is no Storage connected to the CloudManager.
@Tkachov Tkachov CLOUD: Fix HTTP response code TODOs 5f9beb7
@Tkachov Tkachov CLOUD: Update DownloadRequest's TODO
We need a way to reopen DumpFile if we want DownloadRequest to support
@Tkachov Tkachov CLOUD: Fix FolderDownloadRequest TODO 758f46d
@sev- @Tkachov sev- GUI: Regenerate themes 409dd27
@sev- @Tkachov sev- GUI: Fix warnings a65682a
@sev- @Tkachov sev- GUI: Fix format warning e114d1a
@sev- @Tkachov sev- CLOUD: Fix format warning 876b861
@sev- @Tkachov sev- CLOUD: JANITORIAL: Fix code formatting d57fca4
@sev- @Tkachov sev- CLOUD: Fix warning eb268cd
@sev- @Tkachov sev- GUI: JANITORIAL: Fix code formatting 53aa0c4
@sev- @Tkachov sev- CLOUD: JANITORIAL: More whitespace fixes f95073f
@Tkachov Tkachov CLOUD: Fix IndexPageHandler warning 7c9912e
@Tkachov Tkachov CLOUD: Updated BoxListDirectoryByIdRequest
It now checks for all keys in JSON to avoid segfaults and prints
warnings if passed keys are missing or have wrong types.
@Tkachov Tkachov CLOUD: Update BoxListDirectoryByIdRequest
It now uses special CurlJsonRequest static methods to check whether JSON
is an object, has a string or integer parameter.
@Tkachov Tkachov CLOUD: Upload ListDirectory Requests
Lots of checks to avoid JSON-related segfaults added.
@Tkachov Tkachov CLOUD: #define all OAuth2/API-related URLs d57e0c8
@Tkachov Tkachov ALL: Fix debug, warning and error usage
Added prefixes, used debug(9).
@Tkachov Tkachov GUI: Make Options dialog stop LocalServer on close
Commit also adds a fix for StorageWizardDialog, where LocalServer was
used even if USE_SDL_NET was undefined.
@Tkachov Tkachov GUI: Fix SDL_Net-related errors
Checked by rebuilding ScummVM without SDL_Net in MinGW.

Also fixes StorageWizardDialog's warning about _stopServerOnClose.
@Tkachov Tkachov TESTBED: Fix a few Cloud warnings b8fae56
@Tkachov Tkachov ALL: Make simpleSaveNames() a MetaEngineFeature
Added it into hasFeature() of all engines which returned `true` in
simpleSaveNames() before.

As mentioned in #788, SCI is not always using simple names, so it
doesn't have such feature now.
@Tkachov Tkachov GUI: Add "Paste" button in StorageWizardDialog
It pastes clipboard contents as code into 8 fields of that dialog.
(Clipboard support works with SDL2 only.)

"Open URL" and "Paste" buttons are placed in the left column under the
picture (because there is no room for 4 buttons in the bottom row).

Commit also adds "dropbox.bmp", which is just a square 115x115 picture.
Such pictures are would be used as Storages logos in that dialog.

In lowres there is no left column, so all 4 buttons are in the same row.
None of them are visible, because they are overflowed. Container has to
be added to continue working on them.
@Tkachov Tkachov GUI: Add Container in StorageWizardDialog
It now looks fine in both 640x400 and 320x200!
@Tkachov Tkachov GUI: Add Storage providers logos
StorageWizardDialog now shows logo of the Storage being connected (in
modern highres theme).
@Tkachov Tkachov GUI: Fix StorageWizardDialog warning
Removed extra comma in the enum.
@Tkachov Tkachov ALL: Move Clipboard support to OSystem
Commit adds kFeatureClipboardSupport. hasTextInClipboard() and

OSystem_SDL has this feature if SDL2 is used.

EditableWidget and StorageWizardDialog use g_system to access clipboard
@Tkachov Tkachov CLOUD: Add KEY/SECRET override code
The following constants must be defined if ENABLE_RELEASE is:
@Tkachov Tkachov CLOUD: Minor TODO fix 0b97aff
@Tkachov Tkachov CLOUD: Update NetworkReadStream
The latter is available in new libcurl (>= 7.32.0) only, thus the former
is added for older versions support.
@Tkachov Tkachov CLOUD: Fix UploadFileClientHandler
A few possible memory leaks about `_contentStream` there.
@Tkachov Tkachov CLOUD: Add JSON-related checks in BoxStorage 9d96d40
@Tkachov Tkachov CLOUD: Update TokenRefreshers
Box's, Google Drive's and OneDrive's token refreshing requests have more
JSON checks now.
@Tkachov Tkachov CLOUD: Update BoxUploadRequest
More JSON checks there.
@Tkachov Tkachov CLOUD: Update Dropbox Requests
Adding more JSON checks there.
@Tkachov Tkachov CLOUD: Update DropboxStorage
JSON checks added.
@Tkachov Tkachov CLOUD: Update DropboxUploadRequest
JSON checks.
@Tkachov Tkachov CLOUD: Update GoogleDriveStorage
More JSON checks in callbacks.
@Tkachov Tkachov CLOUD: Update GoogleDriveUploadRequest
JSON checks in callback.
@Tkachov Tkachov CLOUD: Update OneDrive
Added JSON checks.

New jsonContainsObject() method added to CurlJsonRequest.
@Tkachov Tkachov CLOUD: Update OneDriveUploadRequest
More JSON checks.
@Tkachov Tkachov CLOUD: Update SavesSyncRequest
Add JSON checks in the callback.
@Tkachov Tkachov CLOUD: Fix Requests
Remove unnecessary JSON warnings, fix a few places.
@uruk @Tkachov uruk GUI: Set tooltip of local webserver button according to server state 9665719
@uruk @Tkachov uruk CLOUD: Remove unused removePathHandler(), make addPathHandler() private b68bd78
@uruk @Tkachov uruk CLOUD: Move determineMimeType to ResourceHandler 64b361b
@sev- @Tkachov sev- CLOUD: Fix warnings 0558ba4
@Tkachov Tkachov CLOUD: Add custom User-Agent
Full version is used like in Eugene's Google Analytics stub. Plus, on
PS3 that string contains "PlayStation", and that would be cool to know
that ScummVM+libcurl+PS3 work together.
@Tkachov Tkachov CLOUD: Fix UploadFileClientHandler
It now redirects user on success not only when file was the last field
in the content, but also when it was uploaded already and Handler worked
further to search for more files.
@uruk @Tkachov uruk CLOUD: Use overriden handle() instead of ClientHandlerCallback in pag…
…e handlers

Using a dedicated callback object for this was an unnecessary overhead.
@Tkachov Tkachov CLOUD: Add "minimal mode" in LocalWebserver
StorageWizardDialog now runs LocalWebserver in "minimal mode" for
security reasons. In this mode server uses only those handlers which
state to support it.

There are two handlers which support minimal mode: IndexPageHandler
(which handles `code` requests needed by StorageWizardDialog) and
ResourceHandler (which provides inner resources like `style.css` or
`logo.png` from `` archive).
@Tkachov Tkachov CLOUD: Mark places where path handling is needed dd9e5a9
@Tkachov Tkachov CLOUD: Handle paths in marked places
Paths containing '../' are forbidden to use in Files Manager. There is
also a special inner black list of paths which are not used and a check
that specified path is under "savepath" or "rootpath" (from "cloud"
@Tkachov Tkachov CLOUD: Add GUI for "rootpath" selection
Cloud tab now contains a button to select path, path label and a clear
@Tkachov Tkachov CLOUD: Update handlers
Now if there is no "rootpath" specified, it's not even listed by
FilesPageHandler and ListAjaxHandler. And, of course, not available to
use anywhere else.
@Tkachov Tkachov CLOUD: Use forbidden combinations
I accidentally tried "folder../" instead "folder/../" and understood
that I made "folder../" forbidden too, though it's a valid folder name.
@Tkachov Tkachov CLOUD: Update LocalWebserver
Reader now reads headers into stream, and some checks are added there
and in UploadFileClientHandler, so if headers are too long, they are
treated as bad request.
@uruk @Tkachov uruk CLOUD: Remove unused includes 02a997e
@Tkachov Tkachov TITANIC: Fix OutSaveFile usage ea360ef
@Tkachov Tkachov COMMON: Fix WriteStream::pos() once again
MemoryReadWriteStream now returns int32, not uint32. It actually doesn't
ever return -1 to indicate that an error occured, so uint32 was a better
choice, but that's what is used in WriteStream base class now.

That method is abstract, so that's also why OutSaveFile had to override
uruk commented Aug 25, 2016

Sorry for not stating my opinion on this PR sooner! Somehow I assumed since I am the mentor, everybody automatically thought that I reviewed and approved all of the code before advising Alexander to start the PR... :) Obviously it was not the case, sorry for this false assumption.
So in my opinion the code is in a very good shape and ready for merge. I think the documentation both on our wiki and on Alexander's blog is sufficient and thorough and it'll help future developers to start hacking on this part of ScummVM. I also don't have any concerns regarding the security of the local webserver code that haven't been dealt with so far, thanks to the valuable comments above.
If nobody else has any objections (mainly @sev- and @wjp), I think it's time to merge this PR, before it starts to rot.

sev- commented Aug 30, 2016

Thus, after discussing with @wjp, we're merging this PR.

I do not think it will go into 1.9.0, which will be started soon, but we need to work on it so it ends up in 1.10.0

Tkachov, congratulations!

@sev- sev- merged commit bfbfbd3 into scummvm:master Aug 30, 2016

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Tkachov commented Aug 30, 2016

Hooray! =)

uruk commented Aug 30, 2016

Congratulations Alexander, great job! :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment