question: how to test new AppData files?

[fedelibre]

I'm trying to test a new appdata file for an application and I'd like to install it so it can be searched and found by GNOME Software or similar software centers. Software is using appstreamcli to perform the search?

Perhaps the command appstreamcli search?
Can anyone explain how this command finds the metadata? Where is the metadata database?

For example, how can I check where these data are fetched from:

$ appstreamcli search lilypond | grep Name
Name: GNU Denemo
Name: feta26
Name: Rosegarden

[ximion]

GNOME Software is using it's own method (unlike AppCenter / Discover), but the principle is the same: Date is pulled from the distribution metadata, which is compiled from your metainfo files and possibly other data sources. For Flatpaks this data comes from the Flapak repo.
For distros, the data is in /var/{lib, cache}/app-info or /usr/share/app-info.
If the distribution provides metadata, it will be preferred over locally installed data by AppStream (and appstreamcli in particular) so an app doesn't change its data once it is installed.
If the app is new and doesn't have an equivalent in the distro data, the metainfo file will be read.
You just have to install the file to /usr/share/metainfo to make that happen, the command appstreamcli put <file.metainfo.xml> will do that for you.

You can force appstreamcli to prefer local metadata by default by changing /etc/appstream.conf and setting PreferLocalMetainfoData=true to see how you app looks like in libappstream-based software centers.

Your metadata will never show up anywhere if it isn't valid though. So you should always run appstreamcli validate <file.metainfo.xml> on your metainfo file and make sure it is at the very least error-free (errors will almost always result in your data being ignored). Getting rid of warnings is a good idea too, though ;-)

[fedelibre]

If the app is new and doesn't have an equivalent in the distro data, the metainfo file will be read.
You just have to install the file to /usr/share/metainfo to make that happen, the command appstreamcli put <file.metainfo.xml> will do that for you.

Ok, thanks, it works.
Before opening this issue I tried putting that file in other directories - like /usr/local/share/metainfo or ~/.local/share/metainfo - and didn't work. I could not imagine that it must be in /usr/share/metainfo. This at least is what I experience in Fedora.

It's a shame that GNOME Software uses its own method.
AppCenter is only for ElementaryOS. I don't want to pull all the KDE stuff just to install Discover for testing. I guess I'll have to use a virtual machine.

[fedelibre]

For Flatpaks this data comes from the Flapak repo

I tried also with a local Flatpak repo, as I'm writing also the Flatpak manifest for this app.
But, as I've described in this issue, flatpak search <app-id> doesn't return anything, while flatpak install <app-id> works. This is confusing.

[fedelibre]

@ximion I've now added the appdata file in the repository where I'm sketching the Flatpak manifest, see here.

appstream-compose command is failing.
I've seen this before in the issue I already mentioned.

This is the log:

$ flatpak-builder --force-clean --repo=/home/fede/flatpak/local _build org.frescobaldi.Frescobaldi.yml
[cut cut cut]

========================================================================
Building module appdata in /home/fede/src/flatpak/frescobaldi/.flatpak-builder/build/appdata-6
========================================================================
Running: mkdir -p ${FLATPAK_DEST}/share/metainfo
Running: cp org.frescobaldi.Frescobaldi.appdata.xml ${FLATPAK_DEST}/share/metainfo
Running: appstream-compose --basename=org.frescobaldi.Frescobaldi --prefix=${FLATPAK_DEST} --origin=flatpak org.frescobaldi.Frescobaldi
Processing application org.frescobaldi.Frescobaldi
Error loading AppData file: failed to parse /app/share/metainfo/org.frescobaldi.Frescobaldi.appdata.xml: Error on line 4 char 7: <component> already set '​' and tried to replace with '
  '
Error: module appdata: Processo figlio uscito con codice 1

There's no error on line 4 char 7. In fact the xml file passes the validation.
Any idea of what's going on?

[ximion]

It's a shame that GNOME Software uses its own method.

It is compliant with the specification though, so everything should pretty much work the same (except for it not reading the appstream.conf flag).
At least in the past, GS preferred local metadata unconditionally, not sure though if that's still the case.

I've now added the appdata file in the repository where I'm sketching the Flatpak manifest, see here.

The file looks absolutely fine to me, spec-wise.... BUT:

There's no error on line 4 char 7. In fact the xml file passes the validation.
Any idea of what's going on?

Yes, sadly. The reference implementation, which appstreamcli uses, uses libxml2 internally to parse the XML file, while appstream-glib, which appstream-util uses and apparently flatpak-builder invokes uses GLib's GMarkup to parse XML, which is less forgiving about unicode zero-width space characters which your XML contains.
I cleaned those up in the attached file:
org.frescobaldi.Frescobaldi.appdata.zip
This will work.

At the same time, I think I need to submit a patch to Publican or add a post-processing step to the spec HTML generation to remove these characters - this is probably where you got the from.
I also wonder whether asglib should handle this issue better, but that's one for @hughsie
While zero-width space is - as far as I understand - valid between XML tags and therefore GMarkup may just understand it, it's also a bit ugly to have it there in the first place (so failing to parse it is better? not sure about that...).

[fedelibre]

I cleaned those up in the attached file:
org.frescobaldi.Frescobaldi.appdata.zip
This will work.

Indeed. Thank you very much

[ximion]

Nice :-)

[fedelibre]

@ximion I still have one problem. I've generated the metadata with this command:

sudo appstream-compose --origin=local --basename=frescobaldi --icons-dir=/usr/share/app-info/icons/local frescobaldi

Then my application becomes visible both in Software and appstreamcli.

But appstreamcli prints a warning:

$ appstreamcli search frescobaldi

** (appstreamcli:7660): WARNING **: 23:06:45.170: Found icon of unknown type, skipping it: unknown

** (appstreamcli:7660): WARNING **: 23:06:45.174: Found icon of unknown type, skipping it: unknown

** (appstreamcli:7660): WARNING **: 23:06:45.177: Found icon of unknown type, skipping it: unknown
Identifier: frescobaldi [desktop-application]
Name: Frescobaldi
Summary: LilyPond Music Editor
Homepage: http://frescobaldi.org/

And in Software the app icon and the screenshots are not displayed.

[fedelibre]

After running this command:

sudo appstreamcli refresh-cache --force --verbose

appstreamcli search frescobaldi doesn't print any warning anymore.

GNOME Software still shows the application but its icon is replaced by some fallback icon (cannot see screenshots, version nor license). And I cannot launch the app.
When I try these actions the logs don't show anything, so it's hard to debug for a user.

[fedelibre]

I've decompressed the xml.gz file I created with appstream-compose and used appstreamcli to validate it. IIUC I found a few errors. First, these images should not have the relative path:

<icon type="cached" height="128" width="128">128x128/frescobaldi.png</icon>
<icon type="cached" height="64" width="64">64x64/frescobaldi.png</icon>

Second, the screenshot has an image type set to source, but should it be remote for URLs?

<screenshot type="default">
    <caption>Main editor view with PDF preview</caption>
    <image type="source">http://frescobaldi.org/images/frescobaldi1-en.png</image>
</screenshot>

Finally, it's been added as part of the KDE project even if it's not. It's just a Qt application. Here's the error:

E - frescobaldi.xml:frescobaldi:4
    The component is part of the KDE project, but its ID does not start with KDEs 
    reverse-DNS name ("org.kde").

So I manually removed the project_group tag from the metadata file.

But these changes doesn't have any effect on how the application is displayed in GNOME Software.
I guess I should open an issue on its tracker.

[ximion]

I've decompressed the xml.gz file I created with appstream-compose and used appstreamcli to validate it. IIUC I found a few errors. First, these images should not have the relative path

Indeed. @hughsie do you know what's going on here? Is appstream-compose in some kind of legacy-compatibility mode maybe?

Second, the screenshot has an image type set to source, but should it be remote for URLs?

No, this one is fine. "source" refers to the original image, as opposed to a thumbnail. In retrospect, "original" would have been a less ambiguous name, but well, we can't change it now without breaking the world (and that minimal confusion is also a bad justification for a breaking change).

Finally, it's been added as part of the KDE project even if it's not. It's just a Qt application. Here's the error

Jup, the "KDE" project group is for projects associated directly with the KDE Project.

But these changes doesn't have any effect on how the application is displayed in GNOME Software.
I guess I should open an issue on its tracker.

I think so...
appstreamcli doesn't look in Flatpak directories, so it may not find any icon. You need to try AppCenter ot KDE Discover to verify whether the icons are missing there as well. If they are missing there as well, there is highly likely an issue with your application or the Flatpak repository. If they are shown there, it's highly likely a GNOME Software bug.
Out of curiosity, what does appstreamcli get org.frescobaldi.Frescobaldi --details print for you?

[fedelibre]

But these changes doesn't have any effect on how the application is displayed in GNOME Software.
I guess I should open an issue on its tracker.

I think so...

Opened here.

appstreamcli doesn't look in Flatpak directories, so it may not find any icon. You need to try AppCenter ot KDE Discover to verify whether the icons are missing there as well. If they are missing there as well, there is highly likely an issue with your application or the Flatpak repository. If they are shown there, it's highly likely a GNOME Software bug.
Out of curiosity, what does appstreamcli get org.frescobaldi.Frescobaldi --details print for you?

AppCenter is available only on Elementary OS, right?
I'll test these software centers next Monday.

I made the test above not with the Flatpak app but with a local build (the application is written in python and uses setup.py.. I've installed it in /usr).

Here's the full output:

$ appstreamcli get frescobaldi --details
Identifier: frescobaldi [desktop-application]
Name: Frescobaldi
Summary: LilyPond Music Editor
Homepage: http://frescobaldi.org/
Icon: frescobaldi.png
Description: Frescobaldi is an advanced text editor to edit LilyPond sheet music files. It 
  aims to be powerful, yet lightweight and easy to use.
  
  Features:
   • Powerful text editor with syntax highlighting and automatic completion
   • Music view with advanced Point & Click
   • Midi player to proof-listen LilyPond-generated MIDI files
   • Midi capturing to enter music
   • Powerful Score Wizard to quickly setup a music score
   • Snippet Manager to store and apply text snippets, templates or scripts
   • Use multiple versions of LilyPond, automatically selects the correct version
   • Built-in LilyPond documentation browser and built-in User Guide
   • Smart layout-control functions like coloring specific objects in the PDF
   • MusicXML, Midi and ABC import
   • Modern user interface with configurable colors, fonts and keyboard shortcuts
   • Translated into the following languages: Dutch, English, French, German, 
  Italian, Czech, Russian, Spanish, Galician, Turkish, Polish, Brazillian Portugese and 
  Ukrainian.
  
  Frescobaldi is designed to run on all major operating systems (Linux, Mac OS X 
  and MS Windows). It is named after Girolamo Frescobaldi (1583-1643), an Italian 
  composer of keyboard music in the late Renaissance and early Baroque period.
Default Screenshot URL: http://frescobaldi.org/images/frescobaldi1-en.png
License: GPL-2.0+
Categories: 
 - AudioVideo
 - Music
 - TextEditor
 - Utility
Provided Items: ↓
 MIME types: text/x-lilypond

[fedelibre]

I made the test above not with the Flatpak app but with a local build (the application is written in python and uses setup.py.. I've installed it in /usr).

I've now tried also with a custom RPM package. I was hoping that GNOME Software would have worked better, but I have the same problems listed before: no icon, no screenshots, launch button not working.
It may be related with the appstream-compose command I'm using?

sudo appstream-compose --origin=local --basename=frescobaldi --icons-dir=/usr/share/app-info/icons/local frescobaldi

[fedelibre]

Finally, it's been added as part of the KDE project even if it's not. It's just a Qt application. Here's the error

Jup, the "KDE" project group is for projects associated directly with the KDE Project.

Might be related to the presence of this line in the desktop file?!?

X-KDE-StartupNotify=true

[fedelibre]

You need to try AppCenter ot KDE Discover to verify whether the icons are missing there as well.

I'm now trying Elementary OS.
First of all, Frescobaldi version in repository is already searchable by both appstreamcli and AppCenter, without any tweaking (as needed on Fedora GNOME).
Icon is visible but there are no screenshots, probably because there's no metainfo file.

I wonder how Elementary generates AppStream metadata.
I've checked the yaml files in /usr/share/app-info/yaml but did not find Frescobaldi.

Here's what I see on Elementary:

$ LANG=C appstreamcli get frescobaldi.desktop --details
Identifier: frescobaldi.desktop [desktop-application]
Name: Frescobaldi
Summary: Editor musicale per LilyPond
Package: frescobaldi
Icon: frescobaldi_frescobaldi.png
Description: Frescobaldi is a LilyPond sheet music editor, with the following features:  * 
  Enter LilyPond scores, then build and preview them with a mouseclick  * Powerful 
  text editor with syntax highlighting and automatic completion  * Point-and-click 
  support: click on notes or on error messages to jump to the    corresponding 
  position in the LilyPond file  * Midi player to proof-listen LilyPond-generated MIDI 
  files  * Powerful Score Wizard to quickly setup a music score  * Snippet Manager to 
  store and apply text snippets, templates or scripts  * Use multiple versions of 
  LilyPond, automatically selects the correct version  * Built-in LilyPond 
  documentation browser and built-in Frescobaldi User Guide  * Translated into Dutch, 
  English, French, German, Italian, Czech, Russian,    Spanish, Galician, Turkish, 
  Polish, Brazillian Portuguese and Ukrainian.
  
  Frescobaldi also includes the following music functions:  * Transpose music  * 
  Change music from relative to absolute and vice versa  * Change the language used 
  for note names  * Change the rhythm (double, halve, add/remove dots, copy, paste) 
  etc  * Hyphenate lyrics using word-processor hyphenation dictionaries  * Add 
  spanners, dynamics, articulation easily using the Quick Insert panel  * Update 
  LilyPond syntax using convert-ly, with display of differences
  
  Per poter usare la sillabazione dei testi occorre installare il pacchetto di 
  sillabazione appropriato per la lingua desiderata, ad esempio hyphen-fr per il 
  francese, hyphen-ca per il catalano, ecc.
  
  Frescobaldi ? il successore per Qt4 di LilyKDE: un plugin di Kate per KDE3.
Categories: 
 - AudioVideo
 - Music
 - Utility
 - TextEditor
Provided Items: ?
 MIME types: text/x-lilypond

[ximion]

I wonder how Elementary generates AppStream metadata.
I've checked the yaml files in /usr/share/app-info/yaml but did not find Frescobaldi.

Those are generated when the package is in the archive, so don't worry about that yet :-)

LANG=C appstreamcli get frescobaldi.desktop --details

This is odd... According to your metainfo file, the AppStream-ID of the data is org.frescobaldi.Frescobaldi, so you should actually be able to use LANG=C appstreamcli get org.frescobaldi.Frescobaldi --details. That you still get metadata indicates that the data is fetched from the .desktop file, rather than from the metainfo file, but in that case you wouldn't have a long description. So I am 99% sure that the data you get there is actually generated from the packaging of the distribution, and not from your metainfo file.

Try the get subcommand with the correct ID please, and also maybe check that /etc/appstream.conf is set to prefer metainfo files.
If you still don't get anything, running appstreamcli with the --verbose flag and attaching the output here will help with finding the culprit.

[fedelibre]

@ximion Sorry for the late reply, I was too busy.
My last comment was probably confusing. I was not testing my "custom package" yet; I was just saying that default behaviour in other software centers is different.

Anyway, appstreamcli is very clear and it works.
I'm having hard times with GNOME Software, but it doesn't matter. I'm sure that everything will work when the application is in Flathub. I don't care for testing anymore.