Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Plugin Support #7

Open
antony-jr opened this issue Nov 10, 2018 · 41 comments

Comments

Projects
None yet
2 participants
@antony-jr
Copy link
Owner

commented Nov 10, 2018

CC : @probonopd

As you insisted on qt plugins instead of a library , I gave plugin support for this code which resides here in a different repo since some may want to use this as a library too(For those who dislike qt plugins).

Now the cool part about plugins is that you don't need to compile any source files of the updater as a developer and this reduces a lot of friction in the development ,however you atleast need a header file to describe the interface.

Another cool thing about plugins is that , plugins built using a major version can be used on all same major version of qt such that the minor version of the built must be less than or equal to the host minor version. Different major versions does not support at all. So we just need to pre-build a single version of plugin with Qt 5.0.0 to support all Qt 5.x.x and Qt 4.0.0 to support all Qt 4.x.x and so on.

So on prebuilt versions we need to distribute two files , a header file and a .so file which is the plugin itself and has to be placed inside the plugin dir relative to the application.

Here is a small video on the plugin -> https://www.youtube.com/watch?v=lhUsjbY0wEY&feature=youtu.be

You can get the prebuilt plugin from the releases.

@probonopd

This comment has been minimized.

Copy link

commented Nov 10, 2018

This is very exciting @antony-jr. Can we make the plugin show up in the application's menu without having to (re-)compile the application?

@probonopd

This comment has been minimized.

Copy link

commented Nov 10, 2018

Can you show in a video how the end result looks in the application's GUI?

@antony-jr

This comment has been minimized.

Copy link
Owner Author

commented Nov 11, 2018

Can we make the plugin show up in the application's menu without having to (re-)compile the application?

Well the plugin has to be loaded using QPluginLoader which returns a QObject to the interface which in turn returns a QObject which is the Delta Revisioner. (Which is then used for updating stuff.)
In case of Qt's internal plugins , Qt sources already has the code to check for the plugin and load it(So When you place the plugin near your application it automatically detects the new plugin and then uses it automatically).
In our case , its third party and thus we have to load it ourselves and then provide a friendly API to
communicate with the plugin. This is all done in the header.
So the re-compiling is just adding code to load the plugin. , Therefore it is impossible without any code change. (Even in case of sparkle framework you need some code to get it running.)

The whole point of plugin is that we don't have to compile heavy source files and we can distribute prebuilt libraries in a plug-in architecture.

In conclusion we can make it show up in the applications menu but we still require some minimal code to load the plugin and show it in the menu.

We can also use the same plugin in PyQt (or any programming language which supports Qt) too without changing the (plugin's) code base but we still require some python code to load and use the plugin.

The developer can also go vanilla and implement his own code to load and use the plugin.

EDIT: I will show you the end result soon , Still this library needs some refactoring and optimization.

@probonopd

This comment has been minimized.

Copy link

commented Nov 11, 2018

Thanks, I understand. Still wondering how some plugins can be used without changing the application's source code or recompiling the application (e.g., themes).

@antony-jr

This comment has been minimized.

Copy link
Owner Author

commented Nov 11, 2018

Still wondering how some plugins can be used without changing the application's source code or recompiling the application (e.g., themes).

I think there has to be something going on , Its more like games , it looks for a dynamic library and uses anything with the same function name and therefore developers don't have to recompile the entire source if they just want to fix that function. This applies to our plugin too.. If the plugin has some error the developer don't have to recompile , He simply needs to replace the plugin file(.so file) thus the development is much more fluid. (i.e) Once he has the code to load the plugin , he does not need to recompile just for a fix in AppImageUpdaterBridge , he just downloads the fixed prebuilt plugin and distributes it to the users. This how I comprehend. I think this is way more good than using a library.

@probonopd

This comment has been minimized.

Copy link

commented Nov 11, 2018

Yes. Even better would be if we could just take an already-compiled application, put in the plugin and maybe a configuration file, and be done. Without changing or compiling the application. You see what I mean?

@antony-jr

This comment has been minimized.

Copy link
Owner Author

commented Nov 11, 2018

@probonopd Finally found a way to automatically execute the updater when our plugin is found nearby without any code addition , you just have to swap in the plugin into your plugin directory in the AppImage.
Its kind of unorthodox but if you are okay then we can use this technique (kind of a hack.)

The developer just have to place two plugins into his/her plugins directory when bundling the AppImage ,
one is libqxcb.so(modified version) and another is libAppImageUpdaterBridgePlugin.so (this is our plugin) , And alas! updating is enabled for that application.

What we do here is that we compile a custom version of the xcb plugin from the qt sources , what we modify is the create method of the plugin interface , in the create method we load our plugin and then start the updater , the updater will only start when the event loop is idle (just for the end user sake). Now qt will automatically execute the create method along with our loading and starting of our updater.

I've already tested this in my local build. But Are you okay with this ? (because its kind of hacking)

@probonopd

This comment has been minimized.

Copy link

commented Nov 11, 2018

The developer just have to place two plugins into his/her plugins directory when bundling the AppImage ,
one is libqxcb.so(modified version) and another is libAppImageUpdaterBridgePlugin.so (this is our plugin) , And alas! updating is enabled for that application.

That is awesome. But why do we need a modified version of libqxcb.so? That may break any time, e.g., when a new version of Qt comes out.

@antony-jr

This comment has been minimized.

Copy link
Owner Author

commented Nov 11, 2018

But why do we need a modified version of libqxcb.so

We need the modified version to load our plugin and start the updater , Since this plugin will be called automatically by qt on linux on startup of any Qt Application using that framework.

Since we are in a AppImage , it won't break since the version is frozen.

We will give plugins for each Qt Major version automatically built in travis.(its not that hard to maintain it.) We download the (specific , just a few source files required to build the plugin) source files from official upstream just add a single header and some code in the create method(this can be automated using python and make), compile and deploy along with our plugin. I'm sure I can do this. ( so anyone can , its actually quite simple)

The end result in the releases will be a archive containing these two plugin files , and when the developer wants autoupdating , on deploy , just before packing it into a AppImage , we replace the two plugins and then package.

Sounds good , if so I can work on a alpha version of this ?

@probonopd

This comment has been minimized.

Copy link

commented Nov 11, 2018

Can we get "Search for updates..." into the application's menu?

@antony-jr

This comment has been minimized.

Copy link
Owner Author

commented Nov 11, 2018

I think we can. I can only say for sure once we test it.

@antony-jr

This comment has been minimized.

Copy link
Owner Author

commented Nov 11, 2018

Okay then I'm off to build the alpha version of this. Once it is usable , I will post a demo video on how to use it. 🚀

@antony-jr

This comment has been minimized.

Copy link
Owner Author

commented Nov 12, 2018

@probonopd I've built a prototype so far , its clumsy but it does show how it works. But there is a minor problem , We will have to compile the modified xcb plugin for Qt version which we wish to support in order to make it work without any flaws , And the dependencies of AppImageUpdaterBridgePlugin should be packed along with the AppImage(i.e libQt5Network.so.5 , I think this is done automatically by the deploy tool , if the dependencies are already satisfied then no need to pack it). We will have to package some openssl files too.(Its not that hard but does consumes time). So to enable auto updates , You just need to copy files from an archive to your AppDir , I think this can be automated by linuxdeployqt in the future.

For now I've compiled the prototype with Qt5.9.3 which is used widely by Qt AppImages , I've also tested it with a AppImage from the AppImageHub. Take a look at this video to see this in action -> https://www.youtube.com/watch?v=-64Ga5Lv10M&feature=youtu.be

Sorry for the shutters in the video , I have a low powered graphics card.

EDIT:
We also might want to add some config file(maybe json) in order to customize the behavior of the Updater.

@antony-jr

This comment has been minimized.

Copy link
Owner Author

commented Nov 12, 2018

I also got Pext to use the Auto-Updater with zero code change , but it crashed because the library failed to find the path to the AppImage. ( I think this is because of python , this can be fixed too )

[ 2018-11-12T20:23:42 ]  UpdateInformation ( Pext )::    INFO:   setAppImage :  "/home/antonyjr/Developer/injector/Pext" . 
[ 2018-11-12T20:23:42 ]  UpdateInformation ( Pext )::   FATAL:   setAppImage : cannot use a directory as a file. 
Segmentation fault (core dumped)

So if my predictions are corrected we can add this updater to any AppImage which uses Qt some way , all we need to do is to locate the xcb plugin and replace it with our modified one and then pack the Updater plugin with its dependencies.

@probonopd

This comment has been minimized.

Copy link

commented Nov 12, 2018

Do you think you can make it work with the Subsurface AppImage, without recompiling it?

@antony-jr

This comment has been minimized.

Copy link
Owner Author

commented Nov 13, 2018

@probonopd Its actually quite simple , before repacking , make sure the AppImage has libQt5Network.so since that is needed by AppImageUpdaterBridge (as I said , linuxdeployqt should pack all the dependencies but in case of repacking its bit tricky , Therefore its highly recommended to copy the plugin on the very first packaging process , i.e linuxdeployqt)

 $ wget "https://github.com/antony-jr/AppImageUpdaterBridgePlugin/releases/download/continuous/AppImageUpdaterBridgePlugin-Qt-5.9.3.tar.gz" # Get the plugin
 $ tar -xvf AppImageUpdaterBridgePlugin-Qt-5.9.3.tar.gz  # extract
 $ wget "https://github.com/Subsurface-divelog/subsurface/releases/download/continuous/Subsurface-4.8.3-337-g9da3988ec652-x86_64.AppImage" # Get Subsurface AppImage
 $ wget "https://github.com/AppImage/AppImageKit/releases/download/continuous/appimagetool-x86_64.AppImage" # Get AppImageTool
 $ chmod +x Subsurface-4.8.3-337-g9da3988ec652-x86_64.AppImage 
 $ chmod +x appimagetool-x86_64.AppImage
 $ ./Subsurface-4.8.3-337-g9da3988ec652-x86_64.AppImage --appimage-extract
 $ cp -rp plugins/* squashfs-root/usr/plugins/
 $ rm -rf squashfs-root/usr/share/metainfo # no need AppStream
 $ ./appimagetool-x86_64.AppImage -u $(./Subsurface-4.8.3-337-g9da3988ec652-x86_64.AppImage --appimage-updateinfo) squashfs-root
 $ ./Subsurface-x86_64.AppImage # Alas! , Your new AppImage.

Here is the new modified AppImage (Please do remember that the plugin is just a prototype) -> https://down.uploadfiles.io/get/ha5dd ( or https://ufile.io/ha5dd ) ( The link will expire after 30 days.)

Screenshot-20181113-183746.png

Screenshot-20181113-183757.png

Screenshot-20181113-183813.png

Note: I don't know why Qt has xcb as a plugin but it does not do more than just creating a new pointer to QXcbIntegration object and return it. Therefore , even though we provide a old modified version , the host version of libQXcbIntegration.so is still used(i.e Packed with the AppImage) , Thus there will be no hindrance using this modified version.

Side-Note : I do wonder what happens if we override kde's libqxcb.so and make it integrate AppImage's automatically.

@probonopd

This comment has been minimized.

Copy link

commented Nov 13, 2018

On my system, the update is stalled at 99%:

stalled

@antony-jr

This comment has been minimized.

Copy link
Owner Author

commented Nov 14, 2018

Thats a known bug in the library , I need to refactor some code. That can be fixed , its not because of the plugin.

Note: Just retry it should work.

@probonopd

This comment has been minimized.

Copy link

commented Nov 14, 2018

Now comes the 100 points question: Can we add "Check for updates..." to the menu via a plugin?

@antony-jr

This comment has been minimized.

Copy link
Owner Author

commented Nov 14, 2018

Yes we can , But only for QMainWindow , since thats that only thing that provides a convenient way to set menu bars. We can even append it has a new entry. And manipulate the menu to our will. I will implement it after we have a solid updater.

@probonopd

This comment has been minimized.

Copy link

commented Nov 14, 2018

This is really exciting!

@probonopd

This comment has been minimized.

Copy link

commented Feb 9, 2019

Is the issue I reported on Nov 13 still around? Thanks for AppImageUpdaterBridgePlugin, it's really very useful!

@antony-jr

This comment has been minimized.

Copy link
Owner Author

commented Feb 9, 2019

I was a little busy with something else , I will try to polish it as much as possible for the users ASAP. The issue occurs randomly so its hard to pin down ,but when the code gets cleaned significantly I can fix it (That will not take too much time , I hope).

EDIT : Having an internal configuration file such as .json file with the update information should increase the speed of information retrieval , anything is better than mapping an entire AppImage for just a data retrieval. For external application this works okay but internally this is a very big overhead.

@probonopd

This comment has been minimized.

Copy link

commented Feb 9, 2019

I think libappimage has a function for it... we need to keep it outside of the squashfs so that the update information can be changed w/o rebuilding the whole AppImage.

@antony-jr

This comment has been minimized.

Copy link
Owner Author

commented Feb 26, 2019

Could you test the same with this version of AppImage. In case you get server refused connection error then try the update a little later.(too much requests makes github suspicious.)

@probonopd

This comment has been minimized.

Copy link

commented Feb 26, 2019

It is working for me. The percentage jumps back and forth a bit while updating, but it does succeed. 👍

@probonopd

This comment has been minimized.

Copy link

commented Feb 26, 2019

At least it worked once. The second try resulted in this:

Update failed for '/home/me/Downloads/Subsurface-x86_64.AppImage' , Because the newly construct AppImage failed to prove integrity , Try again.

I ended up with a file Subsurface-4.8.4-253-g5033c02a110e-x86_64.AppImage.hXMTJ16824.part.

@probonopd

This comment has been minimized.

Copy link

commented Feb 26, 2019

Third try went though OK.

@antony-jr

This comment has been minimized.

Copy link
Owner Author

commented Feb 27, 2019

@probonopd I think using parallel connection to github triggers something that it screws a response with some invalid data. I think when integrity fails , the library should try three times before giving up and throwing an error. The official library or application does not face this since they use linear request and response using libcurl

EDIT : Another possibility is that the temporary file is constructed prematurely before all the responses were written.

EDIT 2: Found the bug , it was because the responses were downloaded from the internet and when the responses were downloaded fully , it called for the construction but the write calls were still busy and thus the entire file was never constructed and tested. So the SHA1 hash on an incomplete file is obviously invalid.

EDIT 3: The problem is fixed in the latest release of the library but not deployed yet , I will do that later. ( it also fixes the unstable progress caused by multiple signals from writer and block downloader )

@probonopd

This comment has been minimized.

Copy link

commented Feb 27, 2019

Do you know why we get the "jumping back and forth" progress indicator?

@antony-jr

This comment has been minimized.

Copy link
Owner Author

commented Feb 28, 2019

Yes , that has been fixed in the latest release of the library but not yet pushed to the plugin.
EDIT : the new build of subsurface is now released in the same link. https://github.com/antony-jr/AppImageUpdaterBridgePlugin/releases/download/continuous/Subsurface-x86_64.AppImage

@probonopd

This comment has been minimized.

Copy link

commented Feb 28, 2019

It's working well for me @antony-jr, thank you 👍

@antony-jr

This comment has been minimized.

Copy link
Owner Author

commented Mar 1, 2019

@probonopd is it possible to get the path to the current executing appimage without relying on the command line arguments passed ? This is because in some application , like a python program , the appimage path can never be resolved from within the script.

@probonopd

This comment has been minimized.

Copy link

commented Mar 1, 2019

It is stored in the $APPIMAGE environment variable.

@probonopd

This comment has been minimized.

Copy link

commented May 11, 2019

The example in qTox/qTox#5643 (comment) is working well for me. 👍 Great job!

@antony-jr

This comment has been minimized.

Copy link
Owner Author

commented May 12, 2019

If I can compile Qt 5.0.0 from source then this plugin could run in all qt5 versions. But unfortunately Qt 5.0.0 can't compile now , it has so many errors , it depends on C++11 when C++11 was experimental. (2013) So I think I will be maintaining a patched version for Qt 5.0.0(and then Qt 6.0.0 if released) which has a customized platform plugins which can load custom plugins using a configuration. Finally https://github.com/TheFutureShell/updatedeployqt will be used to deploy auto updaters for Qt Application packed with AppImage , Qt Installer or simply released via Github releases. Hence the bridges are used.(its actually plugins) (bridge is the word for plugin in this context)

PS: I don't know if developers would be comfortable to use a modified version of platform plugins. (That's the only thing stopping everyone from using this)

Anyways intense development will start from 18.05.19 for this , I plan to finish everything by a week or less. Thank you for showing interest in this.

EDIT:

The example in qTox/qTox#5643 (comment) is working well for me. +1 Great job!

What example , sorry but none of them use this plugin since qTox uses Qt version 5.1 which is lesser than current compiled version which is 5.9 , so we can't replace it. And e2designer uses the library since the author wanted to customize it to the core.

@probonopd

This comment has been minimized.

Copy link

commented May 12, 2019

I think it's OK to require Qt 5 >= 5.6 or so. The same is true for linuxdeployqt.

@antony-jr

This comment has been minimized.

Copy link
Owner Author

commented May 15, 2019

EDIT: Ignore this message , Its seems that the plugin will only be loaded if you use the image format through QPixmap or some other Qt class which processes your custom image format. So For us , hacking the platform plugin is the only way to get around this. Or I will look around the qt docs for more information.

@probonopd Its seems that modifying the qtxcb plugin was an overkill. Just found out that we can create a dummy high level qt plugin such as imageio plugins which will be automatically loaded by qt itself if found under QT_PLUGIN_PATH. We can inject the updater in the create method of the imageio plugin subclass. Its much easier than building the entire qt from source. Now there is no hindrance in using this solution. I will make a prototype soon.

@probonopd

This comment has been minimized.

Copy link

commented May 16, 2019

Looking forward to it @antony-jr. Sounds great

@antony-jr

This comment has been minimized.

Copy link
Owner Author

commented May 17, 2019

@probonopd I tested the plugin on all Qt version packaged AppImage. It seems that it is only possible to inject code in Qt version 5.6 and above. Also the minor version has to match in order to work without any errors. So I made travis-ci build for Qt5.6 , Qt5.7 , Qt5.8 , Qt5.9 , Qt5.10 , Qt5.11 and Qt5.12 . In this repo's releases (https://github.com/TheFutureShell/QtPluginInjector/releases) you can find the modified qxcb plugin for nearly all qt versions (including Qt5.0.0 , I built that using docker in an old version of ubuntu)

So In conclusion , We can only support for Qt5.6 and above.

@probonopd

This comment has been minimized.

Copy link

commented May 17, 2019

We can only support for Qt5.6 and above.

I think this is acceptable if it is clearly documented. Earlier Qt 5 versions had lots of other issues as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.