Package 4.1.0

[arsenetar]

• Merge remaining PRs Increment version to 4.1.0 #733 & Add Exclusion Filters #705
• Update Version to 4.1.0 Increment version to 4.1.0 #733
• Update Changelog Increment version to 4.1.0 #733
• Package Windows 64 bit
• Package Windows 32 bit
• Package OSX Qt (experimental)
• Package OSX Cocoa
• Package .deb
• Notify Ubuntu PPA maintainers
• Notify Arch Linux maintainers
• Make Github Release
• Update website links for new packages.

[arsenetar]

@glubsy, @eugenesan, & @Baltix have any of you had any luck packaging deb /arch packages with package.py? I was trying to use it for a debian package and it was having issues (during dpkg-buildpackage), was wondering if there have been other issues hidden here for some time or this is something new.

[glubsy]

@arsenetar package.py --arch-pkg works fine for Arch. Although it doesn't create the actual package, for that we rely on makepkg.
If the goal is to create a binary package for Arch (installable with pacman -S package.tar.zst for example), would it be a good idea to call makepkg from package.py to generate such package? I can probably integrate a PKGBUILD for that.

[arsenetar]

@glubsy if you think integrating the arch binary packaging into package.py would make things more streamlined then I think that could be a good addition. I know with the ubuntu/debian we would like it to end up working with things like Ubuntu's PPA system and right now I know there is some work to get that functioning correctly.

[glubsy]

I get the following errors while trying to package under Debian 10 Buster. I assume they are the same errors you got.

dpkg-buildpackage: info: host architecture armhf
 dpkg-source --before-build .
dpkg-source: info: using options from dupeguru-4.0.4~unstable/debian/source/options: --compression=xz
 fakeroot debian/rules clean
dh clean
dh: warning: Compatibility levels before 10 are deprecated (level 9 in use)
   dh_auto_clean
dh_auto_clean: warning: Compatibility levels before 10 are deprecated (level 9 in use)
   dh_clean
dh_clean: warning: Compatibility levels before 10 are deprecated (level 9 in use)
 dpkg-source -b .
dpkg-source: info: using options from dupeguru-4.0.4~unstable/debian/source/options: --compression=xz
dpkg-source: error: can't build with source format '3.0 (native)': native package version may not have a revision                                                                                                   
dpkg-buildpackage: error: dpkg-source -b . subprocess returned exit status 2

[arsenetar]

That is a bit different than what I had, although using Ubuntu, mine did not complain about the compatibility level, although I did get the source format error although I think it continued with just that one, the source package seemed to be able to generate just not the binary package I am going to come back around and look at the deb packaging stuff again later. The error I got just told me there were no files to build, which did not really hint at what was wrong. I ended up playing around with the python stdeb package to see if that had better success and it almost worked aside from apparently an issue in setuptools that does not copy header files causing the modules to fail to build, there appear to be workarounds for that, just have not tried those.

[glubsy]

I managed to get passed the native package version may not have a revision error above by removing the revision number from the changelog entry model template in hscommon.build. This reference issue is where I found the workaround. Another reference here.

diff --git hscommon/build.py hscommon/build.py
index 96f52598..f7994636 100644
--- hscommon/build.py
+++ hscommon/build.py
@@ -295,7 +295,7 @@ def build_debian_changelog(
         return [s.strip() for s in result if s.strip()]
 
     ENTRY_MODEL = (
-        "{pkg} ({version}-1) {distribution}; urgency=low\n\n{changes}\n "
+        "{pkg} ({version}) {distribution}; urgency=low\n\n{changes}\n "
         "-- Virgil Dupras <hsoft@hardcoded.net>  {date}\n\n"
     )
     CHANGE_MODEL = "  * {description}\n"

[glubsy]

Oops, looks like I had introduced yet another bug in the #746 by forgetting to make a directory prior to creating the symbolic link inside. See PR #755 for the fix.

Anyway I think I just managed to build for Debian just fine here on raspberry pi 3, both source and binary packages.

Edit: confirmed, seems to work on raspberry pi Debian Buster, apart from the obsolete Qt5/pyqt5 (version 5.11 currently) which lacks some features that were introduced later like QWeelEvent.position(), hopefully will fix itself with time.

[arsenetar]

@SeaCelo, @fuzzy76, @ryanpcmcquen, & @emilbar I have managed to build what will most likely become the 4.1.0 version of the cocoa OSX version, it "should" run on both Intel and Apple arm macs. It was built targeting OSX 11.1 and from what I can tell should be correctly loading the python framework. I am attaching it here as a 4.0.4 RC3 since the version number has not bumped yet. Due to UI changes on the Qt side for 4.1.0 I think we might end up discontinuing the cocoa version unless there is someone to contribute to the UI upkeep on that side of things. Anyhow the dmg is attached (in a zip since github will not allow dmg). Let me know if this one still has issues or not if you have a chance. I have built non-universal qt versions successfully, however I would like to see if I can get a universal build working before posting those.

dupeGuru_osx_cocoa_4.0.4_RC3.dmg.zip

[arsenetar]

@glubsy Hmm, well that is good news it is packaging on Debian (and on a rpi3 at that). I will have to spin up a Debian VM so I can figure out why I am getting odd errors on Ubuntu, perhaps I'll spin a new Ubuntu VM while I am at it.

[emilbar]

Sadly, I can't tets this version > It was built targeting OSX 11.1
Iam stucked with Mojave (10.4.6) because of my old Mac (Early 2009) (but yet powerfull and capable:)

[SeaCelo]

I have both Intel and M1 macs running 11.1. I’ll test it in the coming days (but I have to move, so it will take a few days) Marcelo

…

On Dec 30, 2020, at 9:14 AM, Emil ***@***.***> wrote:  Sadly, I can't tets this version > It was built targeting OSX 11.1 Iam stucked with Mojave (10.4.6) because of my old Mac (Early 2009) (but yet powerfull and capable:) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[glubsy]

I have a OSX Catalina (10.15) virtual machine available to build packages for that target if needed.

[arsenetar]

Looks like pyinstaller/pyinstaller#5107 is going to cause some issues with packaging the qt based versions at this point for me. So probably have to wait for that to be resolved to build complete packages there.

[SeaCelo]

I have both Intel and M1 macs running 11.1. I’ll test it in the coming days (but I have to move, so it will take a few days) Marcelo
…
On Dec 30, 2020, at 9:14 AM, Emil @.***> wrote:  Sadly, I can't tets this version > It was built targeting OSX 11.1 Iam stucked with Mojave (10.4.6) because of my old Mac (Early 2009) (but yet powerfull and capable:) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

Just tried to run it the first time. Intel Mac (11.1). Dupeguru got stuck for over 2 hours with no progress. Still reporting using CPU, but no progress. I decided to quit because it spawned about 8 new instances and it kept doing it. See picture. Nothing reported in Console, so I don't know why it happened. Let me know if you want me to look at anything in specific. I'm running it again now. Will use an M1 tomorrow to see if the same thing happens.

[SeaCelo]

@SeaCelo, @fuzzy76, @ryanpcmcquen, & @emilbar I have managed to build what will most likely become the 4.1.0 version of the cocoa OSX version, it "should" run on both Intel and Apple arm macs. It was built targeting OSX 11.1 and from what I can tell should be correctly loading the python framework. I am attaching it here as a 4.0.4 RC3 since the version number has not bumped yet. Due to UI changes on the Qt side for 4.1.0 I think we might end up discontinuing the cocoa version unless there is someone to contribute to the UI upkeep on that side of things. Anyhow the dmg is attached (in a zip since github will not allow dmg). Let me know if this one still has issues or not if you have a chance. I have built non-universal qt versions successfully, however I would like to see if I can get a universal build working before posting those.

dupeGuru_osx_cocoa_4.0.4_RC3.dmg.zip

Just a thought, but could you release it without the GUI? We could run it from Terminal and avoid the issues.

[arsenetar]

@SeaCelo what mode did you use? I just tried running it again and it looks like there might be something going on with picture mode. I'll have to look into why that might be behaving that way. If you could try the other modes that would be helpful. I did run standard without issues earlier, just ran music it seems to be fine, picture mode is giving me a similar result to what you are showing. It does appear to be an issue specifically with the cocoa version as the qt version runs picture scans just fine.

[arsenetar]

@SeaCelo, I think I found a way to prevent this issue, testing some photo scans now to make sure they run, no longer seeing multiple instances. @emilbar looks like the lowest version I can target now is 10.9, I think you might have meant 10.14 (based on Mojave being mentioned)? I'll try changing the target to 10.12 (since anything less than that starts producing a bunch of compile warnings). Try getting a new version up here in a bit.

[arsenetar]

@SeaCelo, @emilbar Here is a new version with the fix and targeting macos 10.12. Again should work on both intel and arm macs.
dupeguru_osx_cocoa_4_0_4_RC4.dmg.zip

[SeaCelo]

@SeaCelo, @emilbar Here is a new version with the fix and targeting macos 10.12. Again should work on both intel and arm macs.
dupeguru_osx_cocoa_4_0_4_RC4.dmg.zip

Using RC4, it doesn't improve. I'm using Picture mode (contents). During the scan ("Scanning for duplicates"), as soon as it switches to "Performed 0/6 chunk matches", it spawns all the other instances.

The only thing that pops up in my Console is this:
Dec 30 21:25:11 Sleek com.apple.xpc.launchd[1]: Coalition Cache Hit: app<application.com.hardcoded-software.dupeguru.187491676.187491684(502)> [2788]

Standard mode seems to work fine. Music also.

edit: I don't have the latest QT version to test this.

[arsenetar]

Hmm, I see that I have the issue again now, I had it working as was pretty sure that is what I packaged but clearly it is having the issue here on my end as well. I'll walk through some of the other changes I made to try to fix the issue.

[arsenetar]

Ok so I think I figured out why I thought I had it working, seems like if I build it then run it with the python run.py script then I don't see the issue. If you run the application directly via xcode or after building the package it shows the issue.

Seems to come back to the fact that python's multiprocessing library change to use spawn instead of fork for the pools now in 3.8+. This change seems to be due to issues with fork. I tried a few different things to attempt to prevent the issue (like the one from earlier) but I was not able to find one that worked. I have verified that changing it to use fork does resolve the issue. I have manually modified the file for now to make a build (attached below), will need to look into a more permanent solution. The Qt version seems to have no issues with this due to how the application is structured.

dupeguru_osx_cocoa_4_0_4_RC5.dmg.zip

[emilbar]

@arsenetar - yes I have meant 10.14 version (Mojave). Now it works. Made some fast test. I tried Picture scan (both method) on approx. thousand files, and seems to work OK. I also tried Standard mode (Filename and Contents) and get expected results. Found some GUI issues with results window, but want to make some more testing. Should I send my report here or open another thread because I dont know if it is related to macOS?

[arsenetar]

@emilbar You can open a new thread for the results window issue as although probably related to this particular version probably cleaner to keep it separate.

[SeaCelo]

Tried to run RC5 and immediately got an intermittent error (happened 3 times then disappeared). I think it happened because I aways delete the dupeguru directory inside .../Application Support/. Putting it back fixed it after I restarted the app. When I tried deleting it again, the app created the directory, as it normally does. Weird error. Adding it here in case it is important, but the app is working fine now.

dupeguru_error.txt

Like I said, it was a weird error that seems to have disappeared.

Happy New Year!

Edit: I'll try on the M1 Mac sometime tomorrow

[SeaCelo]

Works nearly flawless. The only error I can elicit is when running dupeguru a second time immediately following a successful run. This is the error I get, using Picture mode (contents).
context_set_error.txt

If I close the app and restart it, I can run dupeguru again.

I still didn't manage to make time to test on Intel, but that is my next priority.

[arsenetar]

@SeaCelo I know why that is happening, I will attempt to get another build up here with that issue corrected.

[arsenetar]

@SeaCelo Here is a new version that should have that error fixed. I did notice in this version if you do a scan of the same type after a scan completes and do not close the results window it will throw an error, I need to look into this more but I think it might be happening due to changes for the behavior with the Qt UI which the cocoa version is out of sync with. As long as the results window is closed before starting another scan it seems to work.

dupeguru_osx_cocoa_4_0_4_RC6.dmg.zip

[glubsy]

if you do a scan of the same type after a scan completes and do not close the results window it will throw an error

Although the exact error would probably confirm my assumptions, I believe this happens as expected due to the fact that the cocoa branch doesn't include commits 3382bd5 26d1894 584e9c9. These fixed some GUI logic issues.

If you create a separate branch for the cocoa repository to target as a sub-module (here), I can cherry pick these commits and make sure these fixes are applied without importing the rest of the exclusion list feature. This would be a decent workaround until then.

[arsenetar]

@glubsy not sure if this confirms it but the error originates here:

dupeguru/core/gui/stats_label.py

Line 21 in fd0adc7

self.view.refresh()

Error is basically "NoneType has no method refresh", paraphrasing as I don't have it up right now. I will probably setup a branch for this temporarily if that seems to be the case.

[glubsy]

@arsenetar ah, this is a different issue then.

Still, it might be a good idea to setup a branch for cocoa to target. I'll make a PR when it's up.

[SeaCelo]

@SeaCelo Here is a new version that should have that error fixed. I did notice in this version if you do a scan of the same type after a scan completes and do not close the results window it will throw an error, I need to look into this more but I think it might be happening due to changes for the behavior with the Qt UI which the cocoa version is out of sync with. As long as the results window is closed before starting another scan it seems to work.

dupeguru_osx_cocoa_4_0_4_RC6.dmg.zip

Today I did a full stress test of RC6 in picture mode and I couldn’t get it to throw an error. I ran it on a small set, then ran it again without closing the results window or restarting the app. Then I ran it immediately on 50k pictures. No errors, no problems. I still owe you the M1 test (tomorrow for sure). Let me know if you want me to test anything else.

[SeaCelo]

I just tested RC6 on a M1 MacBook Air. It ran without a hitch. I couldn't make it throw up any errors. I'll try to do a speed test sometime soon, just for fun.

[emilbar]

Finaly I am sending my observations regarding some GUI issues on Mac with dupeguru_osx_cocoa_4_0_4_RC6 version. My english is not so good so I hope that you will understand what I mean.

---

1. Minor problem
I had a problem with columns width in dupeGuru Results window. Width of individual column was to far big. But after deleting old preferences from previous dupeGuru version it is OK now. It was propably due to very long path for Folder column in my previous search.

Suggestion: Calculate column width based on longest name in first two column of Result window (Filename, Folder). I know you can double click on separator line between columns to adjust width, but less clicking is always good :). Although is not that important.

---

2. Improvement
No matter what, if I close all Results window or not main app always ask "You have unsaved results, do you really want to continue?".
My expected behavior would be:

• if all Result windows are closed, dont ask me anything (I already decided that I don't want to save result)
• If I chose YES for answer main app will close old result and show new one. But instead it always just open new results window and all Results windows have a same name, so its dificult to distinguish between them.

Also when opening saved Results, window for that results shows only generic name "dupeGuru Results".

It would be nice if you can assign different name for each new result window. For unsaved window eg. untitled-1, untitled-2 ... or something from search queries Picture Contents - 1, Picture EXIF ...
For saved results of course the name of window should be the name of the filename.

---

3. Bug?

On Mac its possible to set that new documents opens in tabs, and I like this way.

But with dupeGuru in this mode, results in previous Result window just disappear.

Picture 1: First scan with results

Picture 2: new tab after new scan

Picture 3 shows what happens with first tab - empty lines

If I don't use tabs all Result windows shows normal results and when I try to merge them with "Merge All Windows" from Window menu command it happens again.

---

4. AttributeError: 'NoneType' object has no attribute 'refresh'

from arsenetar:
Here is a new version that should have that error fixed. I did notice in this version if you do a scan of the same type after a scan completes and do not close the results window it will throw an error,

I sometimes noticed that error if I try to select "Start Duplicate Scan" from Action menu in all versions that you sent to us. Now I know how to avoide that. Hope you will resolve this error.

---

5. A little bit late: Happy new year!

[arsenetar]

@SeaCelo and @emilbar Thanks for all the feedback on the macOS version. I'll be looking into some of these to see if they can be cleaned up before the 4.1.0 release. Regardless looks like I'll be getting at least a cocoa version out this time round.

[arsenetar]

@glubsy I have tagged 4.1.0 and have a github release for it, so you should be good to update the arch packages

@eugenesan & @Baltix I don't know if you guys are still up for it but the PPA could be updated at this point for 4.1.0. The deb packages build fine on Debian 10 but I have not had a chance to confirm Ubuntu builds.

Windows builds are live at the github release, and the macOS cocoa version should follow shortly. I need to check again, but I don't think I can get a packaged Qt version although it does work on macOS.

[arsenetar]

Found a windows issue with translations #760, fixed that and I was just going to rebuild the windows versions but it seems like the translation files were quite out of date. Updated those, some of the updates fix other broken translations so I think I am going to move to a 4.1.1 release with updated translations files. So @glubsy @eugenesan @Baltix probably best to wait for the 4.1.1 tag now.

[arsenetar]

Closing this to be taken over by #761.