Evaluate porting to Linux

[digitalethics]

I'm not sure how many GNU/Linux users there are but I have long moved on from Windows to using Linux-based operating systems exclusively for all my academic work.

Is there any interest in making Qiqqa available for GNU/Linux systems? I believe this to be a win-win situation as this would bring in a large community of open source contributors who are willing to contribute back and in turn strengthen the overall development of this project.

[GerHobbelt]

I have been considering it, but it requires a significant refactoring or a port / replacement of the commercial wpf ui control library used.

So, yes, I would like to have unix version along with the windows one. It is not my top priority, while I realize the potential "market" there.

As a stopgap measure, we might consider supporting Wine?

[digitalethics]

replacement of the commercial wpf ui control library used.

Qt might be a good cross-platform replacement for the user interface even though I prefer the look of GTK applications. There seems to be some progress to get GTK applications to run under Windows via a subsystem, and I think this would be a more viable way to go rather than trying to run Qiqqa under Linux via Wine or Crossover.

Edit:
I think my last sentence is a bit out of context given that the majority of Qiqqa users are probably running Windows.

@GerHobbelt What do you think is the best way forward with this?

[digitalethics]

Another idea instead of Qt and GTK would be to use V UI:

V UI is a cross-platform UI toolkit written in the V programming language for Windows, macOS, Linux, and soon Android, iOS and the web (JS/WASM). V UI uses native widgets on Windows and macOS, on all other platforms the widgets are drawn by V UI. (...) On Linux, V UI will be a full-featured lightweight alternative to GTK and Qt.

[ljlueloff]

@digitalethics, I think your last sentence was not a bad idea at least for an interim solution...

and I think this would be a more viable way to go rather than trying to run Qiqqa under Linux via Wine or Crossover.

Edit:
I think my last sentence is a bit out of context given that the majority of Qiqqa users are probably running Windows.

Me, what I would really love to be able to do is set it up on my Jetson Xavier AGX (not sure how also adding it ARM support would screw with things), but if it could get to working on Wine in the interim whether it gets ported or not, may help and might even get a larger Linux backing via that method.

According to Wine HQ's appdb https://appdb.winehq.org/objectManager.php?sClass=version&iId=27085&iTestingId=92307 it at least installs on Wine/MacOS, but doesn't run. Ubuntu though, it doesn't even install... What would be the level of effort to solve the Wine compatibility issue vs a full on port? If fixing compatibility with Wine would be easier, it might be worth doing. It would help provide metrics on the non-Windows environment user-base which would be helpful in prioritizing the porting of it, but also may attract experts (definitely not me...if there was some python in it, maybe, but I'm worthless beyond that) that would be interested in porting it once they have it on their preferred OS. Ultimately though, I think it would be a great way to determine if the LOE for the port is warranted...

Just a thought...whether it is a coherent one is an entirely different subject.

[GerHobbelt]

FYI: https://ccifra.github.io/PortingWPFAppsToLinux/Overview.html

QT vs GTK is not an issue here, as Qiqqa is a .NET application and porting it to another language than C# is not on my bucket list as that effort will be huge: that would be the equivalent of a total rewrite from scratch. Unless someone else wishes to take on the entire UI dev work on native Linux, of course.

To get a sense of where you're at:

Check out the blog article I linked above: that guy is in exactly the same boat as we are (plus we have some more issues):

• Qiqqa has several native libraries it uses for core functionality, so his mention of P/Invoke peculiarities is very relevant: interfaces to the PDF handling libs, for instance, will have to be re-done.

• extra: Qiqqa uses a commercial native lib for PDF rendering (SORAX); Qiqqa uses mupdf in another place for text extraction. I've looked at mupdf and am considering using that one for rendering as well, as that removes one commercial lib and would potentially enable us to port Qiqqa to Linux without commercial hassles (paid developer licenses)

  • also note comment by Jimme a while ago in another issue here about SORAX; it's too late right now and I'm dead tired, so search that please. The mupdf work is only partly done (I did a recompile and a few bits to the mupdf code to make it fly for me, but the C#/.NET interface needs some serious work to make it production ready as it's extremely vulnerable to failures originating from the native code -- dig in and you'll see exception hadling done in the mupdf C libraries using the only way that is doable in C: setjmp/longjmp. Meanwhile the .NET interface layer code doesn't "catch" those exceptions which will result in guaranteed crashes for anything out of the ordinary and my library already has quite a few oddball PDFs so that part alone has my work cut out for me, unless someone else is willing and able to tackle the mupdf .NET interface layer code.

    (TODO: link to my mupdf clone/fork repo on github)

• Qiqqa also uses another commercial lib for UI: SyncFunction (if I write that name correctly). Again, you'll need a dev license for that one (paid) and I never checked if SynFunction has a Linux version as I never had to use it for that -- never done UI dev on Linux and frankly I'd rather not become an expert in Qt/GTK/whatever: that's for someone else. I happen to have access to SynFunction for another project where I provide support and as long as I am around that one I'll be able to update (Qiqqa went from 14.x to 16.x or 17.x, whatever it was, in Q4 2019, thank those those folks, but after that access is gone, it's dead and I intend to kick it out as it's a bother that's otherwise useless for me and I'm not willing to shell out the $$$ for that one for Qiqqa alone.

  • Note that another very important native library in use is CEF: that one is Embedded Chrome which comes as a partly native, partly .NET release and again: I never checked if that one runs as-is on Linux. Given the next item in the list below... Hmmmmm...

• Qiqqa is .NET Framework 4.x based, not .NET Core, so that port has to happen first, before you can seriously consider porting it to Linux. See the blog post linked at the top.

  (BTW: I did a quick solution/projects move to .NetCore quite a while ago: it threw a tantrum (as in: a zillion compiler errors) and I didn't look any further than that because I was not interested in pursuing it further as my main focus was (and is) to get a tool that works for large libraries on Windows. Possibly with a nice UI, but that'll be done in the same tech as the existing UI if it's up to me: not because I love it, but I don't want to take on yet another UI framework for a single application. Given that my time is limited, at least until 31st August 2020 (and I pray the shite that takes my time is done by then, because right now it leaves just enough energy for some dumb screen-staring and bike-shedding, not much more 😢 )

TL;DR

• Qiqqa is C#, thus .NET: moving it to another language is not even on my sky's-the-limit horizon.
• Qiqqa is WPF (which is C# based): while I don't like XAML (because I had to learn it for this project and I'd rather stick with web tech, but the Embedded Chrome requirement (Qiqqa Sniffer! Yay!) is keeping me from moving there), so any suggestion is great, as long at it does WPF. Or you'll have to redo the entire UI functionality from scratch, which, I guess, won't have many takers. 😉
• Qiqqa uses two commercial libraries, both .NET and native (AFAICT), so those need to be replaced or paid for (every developer needs a license AFAICT, if they want access to the full thing, instead of just run with the DLLs as-is and pedal on the prayer wheels. I have not investigated their availability or support for Unix or Apple/MAC -- previous experiences with commercial libraries has not left me a fan of any in general, unless they come with full disclosed and user-recompilable source code. (Heed these words; that's 30-odd years of pain encoded in a single sub-sentence. 😉
• Qiqqa uses at least one native library for crucial core functionality (SORAX) and iff and when I happen to replace that one, the only viable candidate is another native library (mupdf): I checked out GhostScript (insert horrible scream here -- it kind of worked, but then turned on me and bit me in my unmentionables: somehow it's grown critically unstable on my box and I don't know what I did to it to make it so. Besides, there were a few oddities when attempting to render single pages, which is a required functionality for Qiqqa, while I was testing the latest GS commandline. Some crap I have to check in my koff koff well-kept koff koff dev notes when I was checking GS for viability - I was willing to pay the price for executable invocation overhead. I was not willing to pay the price of increased flakyness in buggy interface layers. Something with the changed security model in GS 9.51/9.52 IIRC. (🤔 I'm not sure if I did it in this repo, but that trial and fail is stored in a github repo of mine... 🤔 )

Anyway, that's my thoughts on the matter right now.

Apologies if it reads like a braindump mess (it is): it's late, I need to hit the sac to ensure my 4 hours of sleep tonight are a not-too-fanciful potential reality 😉 and y'all be best off checking the blog post linked at the top of this message (I googled it, quickscanned the content and it's at least 90% on the money when considering Qiqqa on Linux) and then go and check out the first, say, 35 github issues for qiqqa and read those, plus, messages, in detail, to grok what's gonna hit you when you wish to move this baby to another platform or upgrade it to CEF (currently Qiqqa is using an, ah, antiquated Mozilla embedded web browser and the qiqqa open issues include several which point out the trouble with that one, where the only replacement/update/upgrade candidate is CEF, as far as I can tell from previous tests and trials with CEF and some other "embedded web browser components": if you want a chance to have Google Scholar working again, you're stuck with CEF. Everyone else failed miserably in my tests at least.

Signing off, hope you cope with this blurb.

[GerHobbelt]

Couldn't let go... Here's:

• the write-up I did about the R&D on PDF rendering to replace SORAX with something open-source (or at least full-source disclosure and user-recompilable -- a fundamental and non-negotiable requirement of mine: there's been too much trouble with binary-only materials in my past life 😡 ): https://github.com/jimmejardine/qiqqa-open-source/blob/master/docs-src/Progress%20In%20Development%20-%20Towards%20Migrating%20the%20PDF%20Viewer%20%2B%20Renderer%20(%2B%20Text%20Extractor).md

• the SORAX native library suggested replacement lib: another native one as mentioned in my message above: mupdf: repo at https://github.com/ArtifexSoftware/mupdf (note this is a clone by the company itself; also note their issue tracker and the issue I filed, which is, IMO, relevant for Qiqqa: https://bugs.ghostscript.com/show_bug.cgi?id=701945, plus the rest of the to-do notes in that doc-src/ blurb I wrote and linked just above). My own work/fork is at https://github.com/GerHobbelt/mupdf

• the commercial lib I was talking about is Syncfusion Essential Studio (thus that was TWO typos for the price of one in my previous message 😲😞 ) at https://www.syncfusion.com/wpf-ui-controls : that's a bundle of WPF custom controls, several of which are in use by Qiqqa. Replacing that one means re-doing the list panel IIRC and possibly a few other UI parts of qiqqa, but it's all WPF enhancement, one way or another. Qt / GTK / xxx is not a viable answer AFAICT, since those don't mix/use WPF, hence imply a much more intensive migration path. Again, caveat: I did not check, this is me extending some old knowledge of mine from my C/C++ heydays into a blunt assumption @ 2020 A.D.

• (Also @jimmejardine, BTW) I just went and checked: SORAX seems DIW: http://www.soraxsoft.com/ seems to be the originator of the PDF render lib in Qiqqa (SORAX Software PDF Reader library) and MS Visual Studio Marketplace doesn't give a hit, while the Goolgy Interwebz produce http://www.soraxsoft.com/ as the company URL via various shareware download sites, where my box reports that URL as unreachable due to DNS issues, which reads to me like an expired/defunct company. Can anyone improve / correct this finding? (Start with Qiqqa's SPDF.DLL, which is the actual native lib we use; it's in /bin/ in the repo -- DLL Properties inspection lists that one as (c) 2011 which is spot one vs the SORAX PDF Reader hits I find on the net, which link to the unreachable http://www.soraxsoft.com/)

[digitalethics]

Thank you for your great reply! This is exactly the kind of information that is needed to get an overview of the project, its current shape and possible directions that could be envisioned. I think before attempting to get a small team together to tackle this in terms of ideas, code and finances, a lot of groundwork has to be done, especially in regards to what you said here:

Qiqqa uses two commercial libraries, both .NET and native (AFAICT), so those need to be replaced or paid for (every developer needs a license AFAICT, if they want access to the full thing, instead of just run with the DLLs as-is and pedal on the prayer wheels. I have not investigated their availability or support for Unix or Apple/MAC -- previous experiences with commercial libraries has not left me a fan of any in general, unless they come with full disclosed and user-recompilable source code. (Heed these words; that's 30-odd years of pain encoded in a single sub-sentence.

I think the best way forward would be that all new code contributions are made with software agnosticism in mind as if we were already preparing for a GNU/Linux version, and secondly, to finish up the process to completely open source the application and remove all commercial libraries.

We can also get started with a list of supporters to see who can contribute what:

List of supporters:
@digitalethics : ideas, finances, UI design, social media

[smartass101]

I succeeded in installing Qiqqa 80 with Wine (both 4.0 and 4.2) on Debian Linux 10. The only trick is that the NET 4.0 runtime needs to be installed separately via winetricks before Qiqqa is installed (which would try to install NET 4.0 alone and fail). Here is what I did:

# recommended to use a separate prefix since the NET runtime will be modified 
export WINEPREFIX="~/.wine_qiqqa"
# during installation this may spit out many errors about context and ResourceMarshal,
# but within a few minutes it apparently succeeds
winetricks dotnet40
# run the setup, should not ask to install NET 4.0 anymore
# note that the build is 32 bit, so wine32 must be available (though WINEARCH can be win64)
wine setup80.exe

Qiqqa then seems to run reasonable well, though the GUI is a little slow to respond and some drop-down menus don't render unless hovered over. When I close Qiqqa, wine reports some crash, but at that point it probably doesn't matter.

Though this worked only on one of my notebooks (an older one), on a newer notebook with a new Intel integrated graphics the welcome window rendered only as black and when I press Esc to start the program it crashes. If I use an emulated Wine desktop the welcome screen renders, but I get the crash when I ask it to start through Esc. Comparing with the log from the other notebook, it seems it crashes just before Initializing SoraxPDFRendererDLLWrapper. So I'm guessing it has something to graphics drivers, haven found a solution yet.

In terms of a native Linux port, I would imagine that since it is written in #C for .NET, it should be quite possible to compile and run it with the open source Mono project, which is multi-platform and sponsored partly by Microsoft.

[smartass101]

I actually tried running it with the Mono runtime (which supports parts .NET 4.0), but sadly some UI DLLs are not provided (if I understood the problem correctly).

Perhaps another alternative is to use the new open-source, cross-platform .NET core project created directly by Microsoft. I tried that as well, but the 5.0 .NET core preview wants some JSON runtime configuration file, whereas the Qiqqa build seems to use some older XML config.

[GerHobbelt]

FYI for visitors: porting to Linux et al is considered by me in my work towards upgrading / dragging the main Qiqqa features into the roaring 20's 😉 but it will be quite some time before Qiqqa will be available on that platform in a native package. 'Till that time, the possible work-around is to use Wine as described by others above.

(The UI is the main showstopper in porting as Qiqqa is WPF based, which is not available in Core .NET or other means to get it to run in native Linux.)

[GerHobbelt]

Note to self (@GerHobbelt) for when I have time to work on this: look into doing Linux installer as an AppImage as per https://appimage.org/

Idea taken from older interview about Synfig: https://opensource.com/article/17/2/opentoonz-2d-animation-software

The point of that exercise is not having to bother with the intricacies of all the distros' package managers, etc.

[digitalethics]

as an AppImage

I'm not entirely sure but it would probably be easier to package the entire Wine runtime inside of a snap package. This would also have added security benefits as you could aim for strict confinement. AppImages are an interesting way of replicating how files were traditionally installed on Windows or macOS systems but I would definitely prefer snaps. From a freedom perspective, the best solution would be flatpaks hosted on Flathub; have to look more into what's the current status regarding Wine.

[Algomorph]

AWS Perhaps someone could look into using the now-open-source Porting Assistant for .NET (originally by AWS) to evaluate better which specific components would need replacement for the full-blown Linux port to become possible. I personally haven't used it, but it could be easier than doing a lot of the work manually.

[Waxwood]

I'd be very pleased to see Qiqqa ported to Linux. I've been following the project for years, especially during my PhD, but never committed to it because it wasn't cross-platform. I suspect that a Linux port would be popular. I will submit a separate request for Qiqqa regarding incorporating features allowing uses to add keywords to sections of text. This would allow Qiqqa to function as a type of Qualitative Data Analysis tool. This would make Qiqqa a uniquely accessible tool with features unavailable in any other open source package.

[GerHobbelt]

Small steps... Moving away for Windows-bound UI (WPF) to HTML :: feasibility tests with CEF / CEFSharp / CEFGlue / Chromely

[GerHobbelt]

Also: Towards migrating the PDF viewer / renderer ( / text extractor). Current v83/v84 bleeding edge (not published yet) has mupdf integrated for PDF viewing, but the text extraction + OCR migration must still be done.

[GerHobbelt]

Xref: discussion #325

[basxto]

QT vs GTK is not an issue here, as Qiqqa is a .NET application and porting it to another language than C# is not on my bucket list

Well, there is Gtk# and Qml.Net https://www.mono-project.com/docs/gui/

Note to self (@GerHobbelt) for when I have time to work on this: look into doing Linux installer as an AppImage as per https://appimage.org/

Idea taken from older interview about Synfig: https://opensource.com/article/17/2/opentoonz-2d-animation-software

The point of that exercise is not having to bother with the intricacies of all the distros' package managers, etc.

Third party packaging will happen as long as it’s compilable and the compilation process can be automated