Add ocDownloader as an option to Nextload?

[e-alfred]

Hello,

you asked on my repository about the API of ocDownloader because you wanted to create an app. I saw your Nextload app and thought that it might be a great idea to add support for ocDownloader to the mobile client app because ocDownloader currently has no mobile app and your app could be expanded to not just downloading Youtube videos.

What do you think about that?

[bbernhard]

Hi,

we could definitely do that!

Although nextload-core already supports other sources besides youtube (it's based on youtube-dl), it's lacking quite a lot of features that ocDownloader has already built-in. So adding another backend to the nextload-client definitely wouldn't hurt ;)

At the moment, the app is pretty limited in it's functionality (mostly because I didn't need more). The only thing that's implemented at the moment is the possibility to create a download job via mobile app. But if more people are using the app, I wouldn't mind extending the application with more functionality.

Do you maybe have an API documentation somewhere (maybe something swagger-based?)? I already looked a bit at the ocDownloader sourcecode and while it's possible to extract all the necessary information from there, it would be great if the information would be easier to extract :)

[e-alfred]

Hello,

ocDownloader also supports Youtube-dl, but currently it is limited to 720p because of the way it is implemented by the former developer (Aria2 and Curl are used for the actual download, youtube-dl only gets the download links). I am trying to change that back so only youtube-dl is used for downloading files, but it is not finished yet.

As for the API, it is quite simple at the moment. There are only three API calls that can be found here: https://github.com/e-alfred/ocdownloader/blob/master/controller/lib/api.php

In detail, these are the calls (Basic Auth is always used as required by Nextcloud itself):

https://localhost/index.php/apps/ocdownloader/api/version?format=json

XHR.setRequestHeader ('OCS-APIREQUEST', 'true');
XHR.setRequestHeader ('Content-type', 'application/x-www-form-urlencoded');
XHR.setRequestHeader ('Authorization', 'Basic ' + btoa(Username + ':' + Passwd));

POST request, currently only checks if the server is active and responds with a HTTP 200 response code, AddonVersion and your numeric version string should be sent in the body of the request. The JSON encoded response should look like this:

{
    "RESULT": true
}

https://localhost/index.php/apps/ocdownloader/api/queue/get?format=json

XHR.setRequestHeader ('OCS-APIREQUEST', 'true');
XHR.setRequestHeader ('Content-type', 'application/x-www-form-urlencoded');
XHR.setRequestHeader ('Authorization', 'Basic ' + btoa(Username + ':' + Passwd));

GET request, this returns a JSON encoded response of all files downloaded by ocDownloader:

{
    "ERROR": false,
    "MESSAGE": null,
    "QUEUE": [
        {
            "GID": "b9e4d3d593e143b0",
            "PROGRESSVAL": 100,
            "PROGRESS": {
                "Message": null,
                "ProgressString": "1.28 MB(100%)",
                "NumSeeders": null,
                "UploadLength": null,
                "Ratio": null
            },
            "STATUS": {
                "Value": "Complete",
                "Seeding": false
            },
            "STATUSID": 0,
            "SPEED": "--",
            "FILENAME": "Setup_23117_x64_free.msi",
            "PROTO": "HTTPS",
            "ISTORRENT": false
        },
        {
            "GID": "3cb8f8676deeff50",
            "PROGRESSVAL": 100,
            "PROGRESS": {
                "Message": null,
                "ProgressString": "1.28 MB(100%)",
                "NumSeeders": null,
                "UploadLength": null,
                "Ratio": null
            },
            "STATUS": {
                "Value": "Complete",
                "Seeding": false
            },
            "STATUSID": 0,
            "SPEED": "--",
            "FILENAME": "Setup_23117_x86_free.msi",
            "PROTO": "HTTPS",
            "ISTORRENT": false
        },
...

https://localhost/index.php/apps/ocdownloader/api/add?format=json

XHR.setRequestHeader ('OCS-APIREQUEST', 'true');
XHR.setRequestHeader ('Content-type', 'application/x-www-form-urlencoded');
XHR.setRequestHeader ('Authorization', 'Basic ' + btoa(Username + ':' + Passwd));

POST request, the URL that should be downloaded by ocDownloader is sent as an encoded URI in the body of the request.

The response is a JSON encoded response:

{
    "ERROR": false,
    "FILENAME": "com.tttdevs.stncbookmarks_1.apk"
}

{
    "ERROR": true,
    "MESSAGE": "InvalidURL"
}

[e-alfred]

I am also interested in creating a desktop client and as Nextload uses QT I think it might be possible to run your code on other platforms as well?

[bbernhard]

Thanks for the short API overview, that was helpful!

I am also interested in creating a desktop client and as Nextload uses QT I think it might be possible to run your code on other platforms as well?

yeah, it runs (with small modifications) on linux, windows, macOSX, android and iOs.

Here's a short preview:

While implementing the ocDownloader backend, I stumbled across this two issues:

• Sometimes I get the following response from the "add" API endpoint
  {"ERROR":true,"MESSAGE":"UnabletoretrievetrueYouTubevideoURL"}

If I wait for 2 seconds and try the exact same API request again, it works. Do you have an idea what that could be? (this happens btw. also with cURL and the ocDownloader webinterface).

• If I add a download task via API request, it doesn't show up in the ocDownloader task list in the webinterface. To rule out, that I am doing something wrong in my application, I've also tried with cURL:

This is the API request:

curl -X POST -d 'URL=https://www.youtube.com/watch?v=km_L0v3C0ms' -H "Content-Type: application/x-www-form-urlencoded" -H "OCS-APIREQUEST: true" -H "Authorization: Bearer wr8Sx-673RD-a8ara-3tgnX-i34Bi"  127.0.0.1:8080/index.php/apps/ocdownloader/api/add?format=json

And that's the response:

{"ERROR":false,"FILENAME":"RAMBO_5_-_LAST_BLOOD_Official_Trailer_2019_Sylvester_Stallone_Action_Movie_HD-km_L0v3C0ms.mp4"}

So, to me it seems successful. But if I look into the ocDownloader webinterface I do not see the video showing up in the downloads list. Any ideas @e-alfred?

[e-alfred]

This looks awesome!

Regarding your questions: Downloads added through the API are not shown in the main window, but if you go to "All Downloads" on the left pane they should be listed there.

The Youtube error is strange, is your Youtube-dl version at the latest release? It gets updated quite frequently because the websites it supports change their code quite frequently and therefore Youtube-dl has to be updated regularly. I usually use the version released through PIP (Python package manager) and a cronjob that updates it frequently.

yeah, it runs (with small modifications) on linux, windows, macOSX, android and iOs.

That would be awesome, can I help you with that somehow?

[bbernhard]

Regarding your questions: Downloads added through the API are not shown in the main window, but if you go to "All Downloads" on the left pane they should be listed there.

aaah, got it. Thanks for the info!

The Youtube error is strange, is your Youtube-dl version at the latest release? It gets updated quite frequently because the websites it supports change their code quite frequently and therefore Youtube-dl has to be updated regularly. I usually use the version released through PIP (Python package manager) and a cronjob that updates it frequently.

Hmm...that's strange. I am pretty sure that I've used the latest youtube-dl binary, but I'll give it another try. In case I stumble across the same issue again, I'll provide you a Dockerfile (basically inherited from the official nextcloud docker image, including aria2 + youtube-dl). Would be great if you could then give it a try too, to see if you experience the same issues (the last time, I could easily reproduce it with docker).

That would be awesome, can I help you with that somehow?

I'll try to build an alpha version in the next days that runs on linux, windows and android (those are the platforms I can easily test). I have a macbook + iOs device too, but I probably need more time to get the macOSX toolchain up and running again :)

After that, it would be great if you could help me a bit with alpha testing :)

[e-alfred]

@bbernhard Do you have any news about a release for testing? I am still interested in a nice desktop/mobile client for ocDownloader.

[bbernhard]

Sorry, completely forgot about that.

I've just pushed the latest changes to the repository. The ocDownloader integration can be found in the ocDownloader branch.

I've also created a nextloud-ocDownloader-docker file which makes testing pretty easy (just spin up a fresh nextcloud instance where ocDownloader is already enabled).

Do you want to compile the app yourself or do you prefer a compiled version?

[e-alfred]

Hello,

I tried to compile your code using QT Creator 4.10.1 and QT 5.13.1 on Windows 10, but I am getting the following error (use of undeclared identifier picker) :

It seems like the File Picker component is missing.

[bbernhard]

That's strange :/

Looks like the error comes from quicknative (see: https://github.com/paulondc/quicknative/blob/master/source/cpp/nativefiledialogext.cpp). I am using the third party dependency quicknative, as Qt doesn't provide a cross platform ImagePicker API yet.

Unfortunately, I do not have Windows 10 installation available for testing (I am still on Windows 7, but need to upgrade soon; as soon as I am on Windows 10, I'll try it there). Just out of interest: Have you tried to compile it for Android?

[e-alfred]

How did you install the quicknative library?

No, I didn't try to build it for Android yet.

[e-alfred]

Could you publish packages for Windows/Linux/MacOS/Android so I can test them? That would be awesome!

[e-alfred]

I was able to compile and run it! But I can't see a place where I can set the username that should be used for login, is it missing?

Also if I enter my Nextcloud URL I get the following TLS error:

qt.network.ssl: QSslSocket::connectToHostEncrypted: TLS initialization failed

[bbernhard]

The easiest way to test the application is probably to use this docker container here: https://github.com/bbernhard/nextcloud-ocdownloader-docker

Regarding the username/password: I decided against the authentication with username+password, as it's always a bit tricky to store the password securely (especially on mobile devices). Therefore, the authentication only works with app tokens. The app token can be generated in the user settings (see https://github.com/bbernhard/nextload-core for some screenshots).

If you are using the above docker container, you should now be able to login with:

url:
http://127.0.0.1:8080

token:
< your generated token >

[e-alfred]

I made it work now, the missing username is just a bit confusing if you use Nextcloud where there is always an account attached to your activities. :-)

Anyway, I found a few small problems in the UI:

• ocDownloader is a generic downloading tool and that works pretty well with Nextload as a client, but there are still some conversion options shown that are not necessary and a bit confusing for ocDownloader (except audio conversion for Youtube downloads). I think it would be better to remove them if ocDownloader is selected as a backend and only showing an audio conversion checkbox (off by default) for Youtube links:

• Maybe a list of existing downloads could be shown? The ocDownloader API already allows that and it would be a nice addon for Nextload I think.

Aside from that, it works pretty well and it would be awesome to have some official releases (on your Github releases page) so this could become the official ocDownloader Desktop/Mobile app!

[bbernhard]

done :)

But not sure yet, how/where I should release it. I mean, I could for sure release it on github, but I doubt that people want to install an apk from an untrusted source..

I also thought about releasing it in the google play store, but I am not sure whether they would approve the app (as it allows you to download youtube videos, they probably will reject the app) :/

[e-alfred]

Nice work! But I am getting the spinner of death if I open the "Progress" tab:

I am getting the following messages in the "Application Output" window in QT Creator:

request:  QUrl("https://server:1111/owncloud/index.php/apps/ocdownloader/api/queue/get?format=json")
Retrying request with uid:  6 ... 500
Retrying request with uid:  6 ... 500
Retrying request with uid:  6 ... 500
Retrying request with uid:  6 ... 500
Retrying request with uid:  6 ... 500
Retrying request with uid:  6 ... 500
Retrying request with uid:  6 ... 500
Retrying request with uid:  6 ... 500
Retrying request with uid:  6 ... 500
Retry max reached...aborting request
Marking all requests as disabled
qrc:/source/qml/items/ProgressItem.qml:49: SyntaxError: JSON.parse: Parse error

Also, it would be great to be able to mark text in the input field so changing/deleting text is easier than it is now.

[e-alfred]

But not sure yet, how/where I should release it. I mean, I could for sure release it on github, but I doubt that people want to install an apk from an untrusted source..

I think for testing it would be good to have at least some releases on the Github releases page. Android would be more difficult, but a release to F-Droid should be possible at least.

[bbernhard]

I am getting the following messages in the "Application Output" window in QT Creator:

That's strange. Unfortunately, I couldn't reproduce the error on my side :/

Could you maybe add console.log(result) on line 49 to the ProgressItem.qml file and re-run the application again? That would help a lot with debugging the issue. Thanks!

[e-alfred]

I found a bug in ocDownloader (JSON data needs to be encoded in UTF-8 and some data doesn't seem to be encoded correctly), so the Nextload Client works without a problem (maybe it should display an error message if an error 500 is detected).

[e-alfred]

Also, it would be good to refresh the progress tab regularly to show the current download status.

[e-alfred]

@bbernhard It would be awesome if we can package the client in binary form for at least Linux, Windows, MacOS and Android. If I can help with something just let me know.

[bbernhard]

Sorry for the late reply! Unfortunately I was pretty busy with other projects the last few weeks. But now with the coronavirus (and the "stay at home policy") I'll try to find some time to continue working on nextload-client.

For me the biggest problem at the moment is the packaging of the application. I think it would be best if the application would be distributed via package managers and "known" distribution channels. So e.g fdroid for android, homebrew for MacOS, deb for Debian/Ubuntu, etc.

Unfortunately, I do not know much about the packaging of applications and to be honest, I probably also don't have enough time to maintain the packages in the long term. So, if you (or anyone else) wants to help with that - any help is really appreciated!

I can of course also add some binaries to the github releases page, but my experience with that is, that the minority of users is comfortable with installing binaries from an untrusted source. So, I think a properly packaged application would be something that's needed (especially as the user group - the nextcloud users - aren't always the most technically savvy people). Another advantage of a dedicated package builder is, that he/she can do some alpha testing too which reduces the risk of releasing a "works on my machine" application.

Although I haven't done any nextload-client development recently, I think I could find enough time in the future to maintain the code in the long term (if there's demand for that). But with the packaging I would definitely need some help from volunteers, as I don't have enough time to do both (too many other open source projects that need some love ;-)).

If someone wants to help with the Windows/Linux/MacOS/Android packaging, please let me know :)

[e-alfred]

I can of course also add some binaries to the github releases page, but my experience with that is, that the minority of users is comfortable with installing binaries from an untrusted source.

I think for testing the app this would suffice for now. Any testing would be good at this stage and most people probably don't want to compile it on their own for various reasons. They still can, of course. I would like to ask some ocDownloader users to try it out then.

For Linux based distributions it might be a good idea to use Flatpak so the effort is simplified to that. For Windows maybe Chocolatey might be an option. To build the packages we have to set up a CI probably, I could look into that if you want.

[bbernhard]

A CI would defintely be great! But I think in the current state it would be even more important to have the basic build scripts in place to manually build/release the app for a specific platform.

Unfortunately, Qt isn't the best framework when it comes to tooling..so there usually aren't that many third party build/packaging helper tools out there for Qt projects. e.g I've tried to get another Qt based app into f-droid store a few months ago, but gave up after a few days. It's not that it isn't doable, but for me personally it wasn't worth the hassle (the f-droid project already has some helper functionality which makes building native Android apps pretty convenient, but they don't have any helper tools for building Qt projects).

So I think before we invest time into setting up a CI job, we need to make sure that we can build the project manually in an automated way. I've started working on a script that (hopefully) generates a linux AppImage. It's not yet finished, but I hope that I maybe can share some code at the end of the week.

For all the other platforms I haven't done anything so far. If you would like to create build scripts for Android/iOs/MacOS or Windows, please feel free to do so. Any help is really appreciated. :)

[e-alfred]

An AppImage would be a great start to make the app more accessible. I will take a look into MacOS and Windows as soon as I can.

[bbernhard]

Awesome!

I am now able to build an AppImage for linux. Here's a first version: https://github.com/bbernhard/nextload-client/releases/download/0.1/Nextload_Client-54fcca6-x86_64.zip

[e-alfred]

Awesome, it works pretty well! Did you add the build script to the repository? Maybe we can move forward with that. I built the app on Windows (again) and it works pretty well too at the moment.

[bbernhard]

I just added the script to the repository. You can find it here: https://github.com/bbernhard/nextload-client/tree/ocdownloader/env/docker/build-appimage

The building of the AppImage is done within the docker image :)

[e-alfred]

Awesome! I am trying to automate compiling on Windows and MacOS, but QT now requires an account to download the SDK which is a bit of a roadblock for others to participate probably.

[bbernhard]

Yeah, that's pretty unfortunate. There was a long discussion on the Qt mailing list about that [1], where they also talked about the CI usecase. I haven't followed the discussion in detail, but I think there are some ways to automate that..but it probably requires quite a bit of work to accomplish that.

I am also not sure how one can automate MacOS/Windows builds in an automated manner? For linux it's pretty easy - there you can pack everything into a docker container. But for Windows/MacOS?

[1] https://lists.qt-project.org/pipermail/development/2020-January/thread.html#38316

[e-alfred]

It is possible, Barrier (a fork of Synergy) and QOwnnotes which are also QT based do it with Azure Pipelines and Gitlab CI for example:

https://github.com/debauchee/barrier
https://github.com/pbek/QOwnNotes

For now compiling manually with QT Creator is quite easy, I now have working releases for MacOS and Windows. I also tried to compile Android releases, but I am currently stuck in a whitescreen after starting the app:

This is what I could find in the adb logcat:

04-02 18:31:45.930 10471 10471 W ResourceType: Invalid package identifier when getting bag for resource number 0x00000000
04-02 18:31:46.148 10471 10471 E Qt      : Can't create main activity
04-02 18:31:46.148 10471 10471 E Qt      : android.content.res.Resources$NotFoundException: String array resource ID #0x0
04-02 18:31:46.148 10471 10471 E Qt      :      at android.content.res.Resources.getStringArray(Resources.java:525)
04-02 18:31:46.148 10471 10471 E Qt      :      at org.qtproject.qt5.android.bindings.QtLoader.startApp(QtLoader.java:423)
04-02 18:31:46.148 10471 10471 E Qt      :      at org.qtproject.qt5.android.bindings.QtActivityLoader.onCreate(QtActivityLoader.java:166)
04-02 18:31:46.148 10471 10471 E Qt      :      at org.qtproject.qt5.android.bindings.QtActivity.onCreateHook(QtActivity.java:266)
04-02 18:31:46.148 10471 10471 E Qt      :      at org.qtproject.qt5.android.bindings.QtActivity.onCreate(QtActivity.java:273)
04-02 18:31:46.148 10471 10471 E Qt      :      at com.liftingcoder.nextload.Main.onCreate(Main.java:52)
04-02 18:31:46.148 10471 10471 E Qt      :      at android.app.Activity.performCreate(Activity.java:6679)
04-02 18:31:46.148 10471 10471 E Qt      :      at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1118)
04-02 18:31:46.148 10471 10471 E Qt      :      at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2618)
04-02 18:31:46.148 10471 10471 E Qt      :      at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2726)
04-02 18:31:46.148 10471 10471 E Qt      :      at android.app.ActivityThread.-wrap12(ActivityThread.java)
04-02 18:31:46.148 10471 10471 E Qt      :      at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1477)
04-02 18:31:46.148 10471 10471 E Qt      :      at android.os.Handler.dispatchMessage(Handler.java:102)
04-02 18:31:46.148 10471 10471 E Qt      :      at android.os.Looper.loop(Looper.java:154)
04-02 18:31:46.148 10471 10471 E Qt      :      at android.app.ActivityThread.main(ActivityThread.java:6119)
04-02 18:31:46.148 10471 10471 E Qt      :      at java.lang.reflect.Method.invoke(Native Method)
04-02 18:31:46.148 10471 10471 E Qt      :      at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:886)
04-02 18:31:46.148 10471 10471 E Qt      :      at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:776)

[bbernhard]

I now have working releases for MacOS and Windows.

very cool!

regarding Android: Which Qt version are you using? The version I am currently using and which works for me is Qt 5.12.6. As Qt 5.12 is a LTS release I think every Qt 5.12.x version should work (at least in theory ;)). With newer Qt versions I am experiencing similar behavior as you've described. I've already mentioned that bug on the Qt mailing list a while ago, but got no response. So as long as Qt 5.12 is supported, I would suggest to stick with that version.

[e-alfred]

Yes, I tried 5.14 just like with all the desktop releases. If you have a version compiled against 5.12, can you share the APK for that?

[bbernhard]

I would be really careful with the Qt version you are using. If with Qt 5.14 everything works for you, then it's fine. My personal recommendation is always to use the latest LTS version of Qt. The folks at Qt try really hard to fix most bugs before the release and try go avoid introducing regressions, but my personal experience with that is, that this unfortunately isn't always the case. (I've experienced it more than once, that a new Qt release introduced a nasty bug). With LTS versions only bug fixes and security patches are added, which makes it less likely to introduce regressions.

regarding android: I am not sure if it's needed, but there's a pretty cool "continuous releasing" service from Microsoft (https://appcenter.ms/). I have used it for a bunch of projects in the past, and it worked really well. It basically works like this: One creates a new project (app) and shares it with other users (testers). As soon as there's a new version of an app, all the testers get a notification (via email). After that, the user opens the Microsoft app on the smartphone to directly download & install the new version of the app. It's mainly used for private alpha/beta testing of apps before the release is uploaded to the google play store. The whole thing is invitation based - so, if you (or other users) are interested in that, I would need your email address to invite you. If that's not needed for now, I can also upload the apk here.

[e-alfred]

This sounds pretty good. You can add this mail adress to the test platform: magictimes@protonmail.com

Maybe we can use Azure Pipelines for building packages too like for example Barrier?

[bbernhard]

I've added you - hope you got an invitation mail. The most convenient way to install the app is via the Visual Studio App Center on your android smartphone: https://play.google.com/store/apps/details?id=com.microsoft.hockeyapp.testerapp&hl=de_AT

regarding Azure Pipelines: I haven't done anything with Azure Pipelines yet (so far my go-to CI has always been Travis CI), but I've heard good things about Azure Pipelines. So this might indeed be an option :)

[e-alfred]

Thanks for adding me, it tried your builds but it seems like the spinner turns indefinitely which I suspect is a missing OpenSSL library (getting TLS initialization errors).

[bbernhard]

Thanks for adding me, it tried your builds but it seems like the spinner turns indefinitely which I suspect is a missing OpenSSL library (getting TLS initialization errors).

I've updated a new version yesterday. Please let me know if that one works.

[e-alfred]

The new build works awesome on Android, but maybe a few error handling and usability changes would be necessary:

• Showing of an error message if a connection to the server fails
• Allowing to swipe between tabs
• Adding a share entry in the Android share menu to make inserting links easier

[e-alfred]

I also tried to compile the app for iOS, but keychainimpl.h is missing it seems.

[bbernhard]

Showing of an error message if a connection to the server fails

should be fixed

Allowing to swipe between tabs

should be fixed

I also tried to compile the app for iOS, but keychainimpl.h is missing it seems.

Unfortunately, Qt doesn't support access to the iOs keychain yet. So I've written my own objective c keychain interface to accomplish that. This keychain interface is something, I've written for another (already abandoned) project, so I am not sure if this still compiles. Need to check that on my macbook.

[e-alfred]

Awesome, the changes work well on all platforms, I also fixed up a few API problems so now Magnet link downloads work with all clients too.

Regarding CI, given that official QT is troublesome to install in a CI environment, I am looking into aqtinstall (https://github.com/miurahr/aqtinstall) on Azure Pipelines (this should help with building on all major desktop OS for now) or AppVeyor which has QT already integrated.

[e-alfred]

I am still not yet finished with the CI stuff, but the Nextcloud App Store now allows to publish "integrations" so I think it should be published there of course.

[31337-4554551n]

I'm just writing here to show love for the idea and that people are still interested.
I would love to be able to use ocdownloader through an app on the phone, hopefully one that can be gotten through fdroid.

I would be useless in terms of making anything happen, but wanted to give kudos/encouragement