Issue #56 - Provide a way to update the darktable lensfun database #57

hfiguiere · 2020-05-06T00:52:21Z

This also update the database.

Closes #56

eszlari · 2020-05-06T03:36:35Z

Why not just copy the db from a more recent upstream git commit, if you want to stay with the old release?

hfiguiere · 2020-05-06T03:40:00Z

From lensfun-update-data

Unfortunately, we have to take into account that the Git database may have a too new format for the locally installed Lensfun.

Hence we shouldn't rely on what's in git.

eszlari · 2020-05-06T04:50:44Z

If you compare the database in both versions (0.3.95 and git HEAD), you see, that all files still have the same version: <lensdatabase version="2"> (https://lensfun.github.io/manual/latest/db_versions.html).

The last version bump was 1 Jan 2016:
lensfun/lensfun@11e3a27

hfiguiere · 2020-05-06T04:53:32Z

But what about the future? What if tomorrow they bump it?

eszlari · 2020-05-06T05:08:40Z

Then we will just switch to git master?

paperdigits · 2020-05-06T15:07:03Z

@eszlari we don't want to buld git master though.

hfiguiere · 2020-05-06T15:09:46Z

Then we will just switch to git master?

No, unless upstream does it.

The lack of recent release of lensfun and the fact that Darktable uses 0.3.95 (which, I read, was not the best idea) is already suboptimal.

eszlari · 2020-05-06T15:51:10Z

Does it actually make a difference in practice? Is it unstable / buggy?

There are other projects that don't like to make releases and where it is no problem to use git master. Also looking at the recent commits: most of them seem to be changes to the database, not the library, which is confirmed by this comment:

While there are many people regularly working on the calibrations, there is currently no active developer for the library. So I have to say AFAIK there is no schedule or plan to make a new release.

bilelmoussaoui · 2020-05-06T17:57:49Z

I don't think that including all the files is a suitable solution imho, why not just bump to the latest working commit for now and see when there's a new release of the library.

nanonyme · 2020-05-06T18:04:13Z

If copying from git, consider adding a simple data validator though which checks version since it's in the XML files. Then you can easily see when compat version changes.

paperdigits · 2020-05-06T18:21:14Z

Can we not specify the zip file as a simple archive in the manifest and unzip it to a specific location?

…

On May 6, 2020 8:09:35 AM PDT, "Hubert Figuière" ***@***.***> wrote: > @eszlari we don't want to buld git master though. No, unless upstream does it. The lack of recent release of lensfun and the fact that Darktable uses 0.3.95 (which, I read, was not the best idea) is already suboptimal. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #57 (comment)

hfiguiere · 2020-05-06T18:28:14Z

Which zip?

The output of lensfun-update-data is a bunch of files, that the update script here will put into a tarball so it is easier to manage with the build process of flatpak. Also the tarball is in the repo. I could avoid checking the db files in, but this is the only reliable way I found to see if there are actual changes to the database since a tarball would be different with different modification time of any of its content. There is no point in doing a new build if the data didn't change.

And lensfun-update-data vs what's in git I'll remind that the former is the sanctionned way to update the data for a distribution without updating the library. Which is why I picked that route.

Now to end the bike shedding, I want the shed colour to be fire engine red.

hfiguiere · 2020-05-06T18:29:18Z

BTW I could just merge the PR right now. I just wanted to make sure this was not done the wrong way.

paperdigits · 2020-05-06T18:30:50Z

I'm sorry you feel this is bikeshedding, I feel like it is a legitimate discussion.

…

On May 6, 2020 11:28:29 AM PDT, "Hubert Figuière" ***@***.***> wrote: Which zip? The output of `lensfun-update-data` is a bunch of files, that the update script here will put into a tarball so it is easier to manage with the build process of flatpak. Also the tarball is in the repo. I could avoid checking the db files in, but this is the only reliable way I found to see if there are actual changes to the database since a tarball would be different with different modification time of any of its content. There is not point in doing a new build if the data didn't change. And `lensfun-update-data` vs what's in git I'll remind that the former is the *sanctionned* way to update the data for a distribution without updating the library. Which is why I picked that route. _Now to end the bike shedding, I want the shed colour to be fire engine red._ -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #57 (comment)

bilelmoussaoui · 2020-05-06T18:33:19Z

BTW I could just merge the PR right now. I just wanted to make sure this was not done the wrong way.

I do think we should wait until we come up with a proper solution instead of just wanting to fix the issue for users. See https://github.com/flathub/flathub/wiki/App-Maintenance#required-files

cc @barthalion @TingPing

paperdigits · 2020-05-06T18:35:03Z

Can we specify lensfun-db.tar.bz2 (the zip file, sorry for the colloquialism) as a source file in the flatpak manifest, then decompress it to the proper place in the flatpak repo?

bilelmoussaoui · 2020-05-06T18:35:43Z

Can we specify lensfun-db.tar.bz2 (the zip file, sorry for the colloquialism) as a source file in the flatpak manifest, then decompress it to the proper place in the flatpak repo?

Yes, that's perfectly doable and flathub-external-checker can update it automatically as well

hfiguiere · 2020-05-06T18:35:45Z

I'm trying to provide the best to have an update database in the flatpak so that:

it's 100% reliably reproductible
we don't break things deliberately like by pulling the master branch of the dependency library
we happen to do the officially sanctionned way by upstream, like a distro would do
it doesn't unecessarily rebuild things

I do not see other alternative even in the suggestion in this thread.

hfiguiere · 2020-05-06T18:37:19Z

Can we specify lensfun-db.tar.bz2 (the zip file, sorry for the colloquialism) as a source file in the flatpak manifest, then decompress it to the proper place in the flatpak repo?

This is already what is being done.

But the proiblem is that the tar.bz2 file was generated by the update script I mention because there is no such thing available otherwise.

hfiguiere · 2020-05-06T18:37:58Z

Can we specify lensfun-db.tar.bz2 (the zip file, sorry for the colloquialism) as a source file in the flatpak manifest, then decompress it to the proper place in the flatpak repo?

Yes, that's perfectly doable and flathub-external-checker can update it automatically as well

Except that this file is not available for download.

paperdigits · 2020-05-06T18:41:04Z

Except that this file is not available for download.

Yes it is: https://wilson.bronger.org/lensfun-db/version_2.tar.bz2

Gnome dictionary does this:

{
            "name": "icons",
            "buildsystem": "simple",
            "build-commands": [
                    "install -Dm644 accessories-dictionary-24.png /app/share/icons/hicolor/24x24/apps/accessories-dictionary.png",
                    "install -Dm644 accessories-dictionary-32.png /app/share/icons/hicolor/32x32/apps/accessories-dictionary.png",
                    "install -Dm644 accessories-dictionary-48.png /app/share/icons/hicolor/48x48/apps/accessories-dictionary.png",
                    "install -Dm644 accessories-dictionary-64.png /app/share/icons/hicolor/64x64/apps/accessories-dictionary.png",
                    "install -Dm644 accessories-dictionary-128.png /app/share/icons/hicolor/128x128/apps/accessories-dictionary.png",
                    "install -Dm644 accessories-dictionary-256.png /app/share/icons/hicolor/256x256/apps/accessories-dictionary.png",
                    "install -Dm644 accessories-dictionary-512.png /app/share/icons/hicolor/512x512/apps/accessories-dictionary.png"
            ],
            "sources": [
                {
                    "type": "file",
                    "path": "accessories-dictionary-24.png"
                },
                {
                    "type": "file",
                    "path": "accessories-dictionary-32.png"
                },
                {
                    "type": "file",
                    "path": "accessories-dictionary-48.png"
                },
                {
                    "type": "file",
                    "path": "accessories-dictionary-64.png"
                },
                {
                    "type": "file",
                    "path": "accessories-dictionary-128.png"
                },
                {
                    "type": "file",
                    "path": "accessories-dictionary-256.png"
                },
                {
                    "type": "file",
                    "path": "accessories-dictionary-512.png"
                }
            ]
        },

Could we do similar?

paperdigits · 2020-05-06T18:47:06Z

Perhaps something like this

{
			"name": "lensfun-db",
			"buildsystem": "simple",
			"sources": [
				{
					"type": "archive",
					"url": "https://wilson.bronger.org/lensfun-db/version_2.tar.bz2",
					"sha256": "something-something"
				}
			],
			"build-commands": [ "cp /extracted-dir/ /correct/path/to/lensfun-db/" ]
		},

hfiguiere · 2020-05-06T18:47:07Z

Except that this file is not available for download.

Yes it is: https://wilson.bronger.org/lensfun-db/version_2.tar.bz2

I didn't find an indication of where to find it (I might have missed it, though)
this is just a tarball for which we don't even know which version it is based on the URL, which mean that expecting to download it and match the sha256 will randomly break the build, and make the build not reproducible since for a specified revision of the git repository for the flatpak the build result could be different.

Gnome dictionary does this:

Did you look at the .json in this PR?

Could we do similar?

You mean copy 60 files by hand, install a tarball?

hfiguiere · 2020-05-06T18:48:14Z

  			"type": "archive",
  			"url": "https://wilson.bronger.org/lensfun-db/version_2.tar.bz2",
  			"sha256": "something-something"

So the day the tarball change, the build fail and there is no way to do it again.

paperdigits · 2020-05-06T18:48:42Z

So the day the tarball change, the build fail and there is no way to do it again.

Yes so we update the sha256 and build again.

bilelmoussaoui · 2020-05-06T18:49:41Z

I'm trying to provide the best to have an update database in the flatpak so that:
* it's 100% reliably reproductible

* we don't break things deliberately like by pulling the master branch of the dependency library

* we happen to do the officially sanctionned way by upstream, like a distro would do

* it doesn't unecessarily rebuild things
I do not see other alternative even in the suggestion in this thread.

  			"type": "archive",
  			"url": "https://wilson.bronger.org/lensfun-db/version_2.tar.bz2",
  			"sha256": "something-something"
So the day the tarball change, the build fail and there is no way to do it again.

That's the exact reason why we have flathub-external-data-checker. If the url of an application that we can't re-distribute directly, the bot updates the URL and their checksums each hour automatically and send/merge the pull requests as well.

hfiguiere · 2020-05-06T18:54:53Z

That's the exact reason why we have flathub-external-data-checker. If the url of an application that we can't re-distribute directly, the bot updates the URL and their checksums each hour automatically and send/merge the pull requests as well.

But this is not the case of something that can't be redistributed. This is something that can be checked, redistributed, etc.

I'll ask the question. What is wrong with the PR? So far I only got told there are alternative for which none of the parameters I listed are fullfilled.

This solution even fits the parameters list on the app maintenance wiki page since this is handling a specific problem for a specific solution. I really wish it had been easier, but I haven't found a better way.

bilelmoussaoui · 2020-05-06T18:57:37Z

That's the exact reason why we have flathub-external-data-checker. If the url of an application that we can't re-distribute directly, the bot updates the URL and their checksums each hour automatically and send/merge the pull requests as well.

But this is not the case of something that can't be redistributed. This is something that can be checked, redistributed, etc.

I'll ask the question. What is wrong with the PR? So far I only got told there are alternative for which none of the parameters I listed are fullfilled.

This solution even fits the parameters list on the app maintenance wiki page since this is handling a specific problem for a specific solution. I really wish it had been easier, but I haven't found a better way.

In such case, you can just generate those files in a separate repository, create a tarball and use it from there if it can be redistributed.

eszlari · 2020-05-06T19:12:17Z

What is wrong with the PR?

It's an over engineered solution to a problem that doesn't (yet) exist.

The manifests on Flathub should strive to be maintainable. What if tomorrow, you are hit by a bus? I think everybody should be able to understand a manifest as easy as possible and submit PRs.

Just use a git commit and add a comment to the manifest like:

/* check lensdatabase version */

hfiguiere · 2020-05-06T19:16:37Z

The manifests on Flathub should strive to be maintainable. What if tomorrow, you are hit by a bus?

They'd be people in deeper trouble than a flathub repository.

Also there is a README.md in the PR for that exact reason. If anything is unclear in it, I'd love to fix it.

Anyway it seems that is consensus that you don't like it. Feel free to reopen reuse if you feel there is anything usable.

Issue #56 - Provide a way to update the darktable lensfun database

00b79ff

hfiguiere closed this May 6, 2020

hfiguiere deleted the lensfundb branch May 13, 2020 12:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue #56 - Provide a way to update the darktable lensfun database #57

Issue #56 - Provide a way to update the darktable lensfun database #57

hfiguiere commented May 6, 2020

eszlari commented May 6, 2020

hfiguiere commented May 6, 2020

eszlari commented May 6, 2020

hfiguiere commented May 6, 2020

eszlari commented May 6, 2020

paperdigits commented May 6, 2020

hfiguiere commented May 6, 2020

eszlari commented May 6, 2020

bilelmoussaoui commented May 6, 2020

nanonyme commented May 6, 2020 •

edited

Loading

paperdigits commented May 6, 2020 via email

hfiguiere commented May 6, 2020 •

edited

Loading

hfiguiere commented May 6, 2020

paperdigits commented May 6, 2020 via email

bilelmoussaoui commented May 6, 2020

paperdigits commented May 6, 2020

bilelmoussaoui commented May 6, 2020

hfiguiere commented May 6, 2020

hfiguiere commented May 6, 2020

hfiguiere commented May 6, 2020

paperdigits commented May 6, 2020

paperdigits commented May 6, 2020

hfiguiere commented May 6, 2020

hfiguiere commented May 6, 2020

paperdigits commented May 6, 2020

bilelmoussaoui commented May 6, 2020

hfiguiere commented May 6, 2020

bilelmoussaoui commented May 6, 2020 •

edited

Loading

eszlari commented May 6, 2020

hfiguiere commented May 6, 2020

Issue #56 - Provide a way to update the darktable lensfun database #57

Issue #56 - Provide a way to update the darktable lensfun database #57

Conversation

hfiguiere commented May 6, 2020

eszlari commented May 6, 2020

hfiguiere commented May 6, 2020

eszlari commented May 6, 2020

hfiguiere commented May 6, 2020

eszlari commented May 6, 2020

paperdigits commented May 6, 2020

hfiguiere commented May 6, 2020

eszlari commented May 6, 2020

bilelmoussaoui commented May 6, 2020

nanonyme commented May 6, 2020 • edited Loading

paperdigits commented May 6, 2020 via email

hfiguiere commented May 6, 2020 • edited Loading

hfiguiere commented May 6, 2020

paperdigits commented May 6, 2020 via email

bilelmoussaoui commented May 6, 2020

paperdigits commented May 6, 2020

bilelmoussaoui commented May 6, 2020

hfiguiere commented May 6, 2020

hfiguiere commented May 6, 2020

hfiguiere commented May 6, 2020

paperdigits commented May 6, 2020

paperdigits commented May 6, 2020

hfiguiere commented May 6, 2020

hfiguiere commented May 6, 2020

paperdigits commented May 6, 2020

bilelmoussaoui commented May 6, 2020

hfiguiere commented May 6, 2020

bilelmoussaoui commented May 6, 2020 • edited Loading

eszlari commented May 6, 2020

hfiguiere commented May 6, 2020

nanonyme commented May 6, 2020 •

edited

Loading

hfiguiere commented May 6, 2020 •

edited

Loading

bilelmoussaoui commented May 6, 2020 •

edited

Loading