WIP: Move the database of supported devices out into runtime loaded f… #293

hughsie · 2017-10-28T22:25:28Z

…iles

When fwupd is installed in long-term support distros it's very hard to backport
new versions as new hardware is released.

There are several reasons why we can't just include the mapping and quirk
information in the AppStream metadata:

The extra data is hugely specific to the installed fwupd plugin versions
The device-id is per-device, and the mapping is usually per-plugin
Often the information is needed before the FuDevice is created
There are security implications in allowing plugins to handle new devices

The idea with FuHwdb is that the end user can drop an additional (or replace
an existing) file in a .d director with a simple format and the hardware will
magically start working. This assumes no new quirks are required, as this would
obviously need code changes, but allows us to get most existing devices working
in an easy way without the user compiling anything.

This allows us to fix issues like https://github.com/hughsie/fwupd/issues/265

FIXMEs:

I'm not happy with FuHwdb as a name, it's not a DB and too similar to FuHwids
Is .hwdb a good extension? Do we need Yet Another File Format? Really?
Could we use a fwupd-installed AppStream file with the quirks in?
org.freedesktop.fwupd.plugin.dfu.quirks.metainfo and then use <custom> and <value> --
XML isn't super human-write-friendly and kinda verbose...
The DFU parts need to be split off as a different commit
Other plugins need to be ported, e.g. ebitdo, unifying, purism, and avr32
This needs to be tested on real hardware

ghost · 2017-10-30T12:03:11Z

This looks good.

For the Jabra DFU quirk handling code to be completely freed of PID switch statements, we would need to be able to also encode the 'rep' and 'adr' bytes of the FWU mode switch command in the quirks file. Then new DFU devices that do not require new quirks can be added without code changes.

superm1

I like the approach. Something else I think that could be interesting is to let the quirks come down with LVFS metadata distributed with updates.
If you allow quirks to be put directly into CAB files, you can in effect not have to even do "OS" updates of quirks for supporting new devices in some cases.

superm1 · 2017-10-30T14:57:04Z

plugins/dfu/dfu-common.c

+dfu_utils_buffer_parse_uint8 (const gchar *data, guint pos)
+{
+	gchar buffer[3];
+	memcpy (buffer, data + pos, 2);


I think you'll want a test whether pos is bigger than the size of the buffer to disallow overflow.

I've removed pos and added a big fat warning to each function now.

superm1 · 2017-10-30T14:57:15Z

plugins/dfu/dfu-common.c

+dfu_utils_buffer_parse_uint16 (const gchar *data, guint pos)
+{
+	gchar buffer[5];
+	memcpy (buffer, data + pos, 4);


superm1 · 2017-10-30T14:57:24Z

plugins/dfu/dfu-common.c

+dfu_utils_buffer_parse_uint32 (const gchar *data, guint pos)
+{
+	gchar buffer[9];
+	memcpy (buffer, data + pos, 8);


superm1 · 2017-10-30T14:59:25Z

plugins/dfu/dfu-device.c

+			priv->quirks |= DFU_DEVICE_QUIRK_IGNORE_UPLOAD;
+		if (g_strcmp0 (split[i], "attach-extra-reset") == 0)
+			priv->quirks |= DFU_DEVICE_QUIRK_ATTACH_EXTRA_RESET;
+	}


On each of these you should be able to continue after assigning a quirk. there should only be one for each split[i].

Thanks, fixed.

superm1 · 2017-10-30T15:05:02Z

src/fu-engine.c

+	/* FIXME: special case DELL, perhaps this can go in the XML?
+	 * com.dell.uefi*.firmware -> AS_VERSION_PARSE_FLAG_NONE */
+	if (g_str_has_prefix (as_app_get_id (app), "com.dell.uefi"))
+		flags = AS_VERSION_PARSE_FLAG_NONE;



When you say maybe this can go in XML, do you mean LVFS output metadata or individual XML for uploads?

I think it's better to keep this directly in fwupd and let the engine parse the quirks like plugins do. Although it's only being done for UEFI plugin I don't think this will necessarily only be a UEFI thing in the future. Everyone can do their own interpretations of versions.

Right, this shouldn't go in the XML I suppose. The problem here is that FuQuirks doesn't support "has_prefix" or any kind of wildcarding, I suppose we need to add that too.

I've added this now.

superm1 · 2017-10-30T15:09:34Z

contrib/fwupd.spec.in

@@ -208,6 +208,8 @@ mkdir -p --mode=0700 $RPM_BUILD_ROOT%{_localstatedir}/lib/fwupd/gnupg
 %{_unitdir}/fwupd.service
 %{_unitdir}/system-update.target.wants/
 %dir %{_localstatedir}/lib/fwupd
+%dir %{_datadir}/fwupd/quirks.d


Just a thought, can rpm packaging let you just wildcards to pick everything like %dir %{_datadir}/fwupd/*? It should let this have to change less when you shuffle stuff around if you can.

Yes, although IIUC it's best practice to list them individually, which means you notice if they go away or get renamed. You could argue the file-list could just be * :)

But considering you're the upstream and you know when files come and go as a result, is that still a relevant best practice? Just seems like paperwork at that point....

I think using logic with the Fedora Packaging Policy is a dangerous game :) The other thing %dir lets you do is choose the permissions or group ownership, although in this case the quirks are not security sensitive and can use the default.

Ah well I guess this is a difference from debian approach when it comes to policy. dh_fixperms will run at set policy on everything unless you override it.

dh_fixperms is a debhelper program that is responsible for setting the permissions of files and directories in package build directories to a sane state -- a state that complies with Debian policy.

I guess you know fedora policy and such better than me on what would fly with this kind of stuff. I just think TPS reports are overrated and try to avoid them when I can.

hughsie · 2017-10-30T22:09:00Z

Okay, lots of fixes. @superm1 if you can test the new Dell stuff that would be handy. I'm also not sure about the prefix names, e.g. do we want:

fwupd-uefi-version-format
FwupdUefiVersionFormat
org.freedesktop.fwupd.plugin.uefi:version-format
org.freedesktop.fwupd.plugin.uefi:VersionFormat
fwupd:plugin:uefi/version_format
something else entirely.

I'm also not sure how to document the quirks; in the code they'd be scattered around and in the .quirk files there might be multiple files adding keys with the same prefix. In a random .txt file it might be tricky to describe what the quirks are doing without the code inline. Ideas welcome.

superm1 · 2017-10-31T18:24:56Z

I think it would be a good practice to put all the quirks used in fu-quirks.h (or a similar header file name), and document them there. That header file can then be installed with fwupd so that you can easily look at it from a text editor to know what quirks were supported by the installed version of fwupd.

To avoid having to do code inline how about every quirk in the header file gets a comment like this:

Standard behavior: Version format is recognized as AA.BB.CCDD
Quirked behavior: Version format is recognized as AA.BB.CC.DD

I think adding pointers (or copying )the code is unnecessary effort. Once you've got a string, it's easy enough to just search from github or grep the codebase to find where it's used.

Other idea is just that ini style file like you've got implemented but same strings above. Everything that is quirkable should have an empty section then by default in the ini file.

Regarding prefix name, nothing else is going to use this other than fwupd right? I think dbus style naming is confusing outside a dbus context. I'm personally a fan of the first option you listed (fwupd-<consumer>-<quirk type>)

When fwupd is installed in long-term support distros it's very hard to backport new versions as new hardware is released. There are several reasons why we can't just include the mapping and quirk information in the AppStream metadata: * The extra data is hugely specific to the installed fwupd plugin versions * The device-id is per-device, and the mapping is usually per-plugin * Often the information is needed before the FuDevice is created * There are security implications in allowing plugins to handle new devices The idea with quirks is that the end user can drop an additional (or replace an existing) file in a .d director with a simple format and the hardware will magically start working. This assumes no new quirks are required, as this would obviously need code changes, but allows us to get most existing devices working in an easy way without the user compiling anything. This allows us to fix issues like https://github.com/hughsie/fwupd/issues/265

hughsie · 2017-10-31T22:00:49Z

I've spent a bunch of time documenting all the quirks and giving them better names. I did put all the documentation in fu-quirks.h like you suggested but rather than disting the file added them to the gtk-doc so they show up in the product docs.

superm1 · 2017-10-31T22:10:43Z

That looks really good, and gtk-doc output is definitely a much better consumer than a header file.

The DFU ones are still missing, is that intentional? I was actually thinking that those are the more interesting ones because there are so many and they can be used together.

superm1 · 2017-10-31T22:12:17Z

plugins/dell/dell.quirk

@@ -0,0 +1,8 @@
+[fwupd-plugin-uefi-version-format]


Should this be FU_QUIRKS_PLUGIN_UEFI_VERSION_FORMAT for consistency?

Well, the #define has to start with FU_QUIRKS to make the gtk-doc work. I guess we could have [fu-quirks-plugin-uefi-version-format] if you think that's clearer?

I was just meaning for consistency that when you search the project you find all the instances of the #define string in the same place. So use them in the .quirk file rather than what they turn into.

I think doing that makes it very complicated indeed. We'd need to have a table of key-values and also generate the fu-quirks.h file too... :(

superm1 · 2017-10-31T22:12:49Z

plugins/dell/dell.quirk

+Dell Inc.=none
+Alienware=none
+
+[fwupd-daemon-version-format]


Same question, FU_QUIRKS_DAEMON_VERSION_FORMAT ?

superm1 · 2017-10-31T22:13:56Z

plugins/dell/meson.build

+install_data(['dell.quirk'],
+  install_dir: join_paths(get_option('datadir'), 'fwupd', 'quirks.d')
+)
+


If those other strings can't work, would pre-processing (.quirks.in ->.quirks) make sense rather than hardcoding in source in two places?

superm1 · 2017-10-31T22:16:32Z

src/fu-quirks.h

+ * Assigns optional quirks to use for a DFU device which does not follow the
+ * DFU 1.0 or 1.1 specification.
+ *
+ * Default value: `none`


So I think this should point to DfuDeviceQuirks in plugins/dfu/dfu-device.h, but those strings aren't really discoverable right now.

I think they probably need documenting (fully) in this gtk-doc block. Maybe in a list?

Yeah that's fine.

superm1 · 2017-10-31T22:18:18Z

src/fu-quirks.h

+ *
+ * Since: 1.0.1
+ */
+#define	FU_QUIRKS_DAEMON_VERSION_FORMAT		"fwupd-daemon-version-format"


I think put the UEFI plugin as the last plugin, and the daemon at the very end. So all plugin quirks come in order (alphabetically) and then daemon quirks come afterwards. Makes them easier to find rather UEFI in between all the DFU ones.

so fwupd-dfu-alternate-vidpid-plugin, fwupd-dfu-jabra-detach-plugin, fwupd-uefi-version-format-plugin and fwupd-version-format-daemon? Doing it this way moves the "where" to right at the end. Maybe we drop the -plugin suffix entirely?

I think drop the suffix entirely yes. Order sounds right.

hughsie · 2017-10-31T22:19:45Z

The dfu ones should be here, I force pushed so you might need to reload this page to avoid the ajax being funky.

This is slightly more verbose than desired as we also have to include the quirk information when running the dfu-tool, which does not have an already set-up FuQuirks object as it has no plugin.

This allows us to remove the Jabra-specific quirk entry in the device bitfield, and more importantly allows us to support some more Jabra devices in the future without code changes.

hughsie · 2017-11-02T10:27:20Z

Okay, all fixes done, thanks for all the help so far.

superm1

LGTM now

hughsie · 2017-11-02T19:11:25Z

@gnolsen when you're ready, could you open a new PR for the new Jabra models please? It should just be a case of adding a few lines to dfu.quirk

hughsie requested a review from superm1 October 28, 2017 22:25

hughsie force-pushed the wip/hughsie/hwdb branch 2 times, most recently from 309dbd4 to 88f236c Compare October 30, 2017 11:11

hughsie force-pushed the wip/hughsie/hwdb branch from 88f236c to 6af3c4d Compare October 30, 2017 13:36

superm1 requested changes Oct 30, 2017

View reviewed changes

hughsie mentioned this pull request Oct 30, 2017

Wip/hughsie/atmel flip #292

Merged

hughsie force-pushed the wip/hughsie/hwdb branch from 6af3c4d to e06ee7e Compare October 30, 2017 22:03

hughsie force-pushed the wip/hughsie/hwdb branch from e06ee7e to 1093cc0 Compare October 31, 2017 22:00

superm1 reviewed Oct 31, 2017

View reviewed changes

hughsie added 5 commits November 2, 2017 10:23

dfu: Use the FuQuirk infrastructure to move the quirks out of the code

95817dc

This is slightly more verbose than desired as we also have to include the quirk information when running the dfu-tool, which does not have an already set-up FuQuirks object as it has no plugin.

dell: Use the new quirk infrastructure for version numbers

e6cf00d

dfu: Move some code to allow re-use

2507f82

dfu: Use FuQuirk to encode the Jabra magic packet contents

5be97d5

This allows us to remove the Jabra-specific quirk entry in the device bitfield, and more importantly allows us to support some more Jabra devices in the future without code changes.

dfu: Use FuQuirk to map the chip ID to the AVR32 UM0424 alt-name

a2ea0e3

hughsie force-pushed the wip/hughsie/hwdb branch from 1093cc0 to a2ea0e3 Compare November 2, 2017 10:26

superm1 approved these changes Nov 2, 2017

View reviewed changes

hughsie merged commit 674ed34 into master Nov 2, 2017

hughsie deleted the wip/hughsie/hwdb branch November 2, 2017 19:08

WIP: Move the database of supported devices out into runtime loaded f… #293

WIP: Move the database of supported devices out into runtime loaded f… #293

Conversation

hughsie commented Oct 28, 2017 • edited Loading

ghost commented Oct 30, 2017

superm1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hughsie commented Oct 30, 2017

superm1 commented Oct 31, 2017

hughsie commented Oct 31, 2017

superm1 commented Oct 31, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hughsie commented Oct 31, 2017

hughsie commented Nov 2, 2017

superm1 left a comment

Choose a reason for hiding this comment

hughsie commented Nov 2, 2017

hughsie commented Oct 28, 2017 •

edited

Loading