eopkg: backend cannot handle quick successive operations

[joebonrichie]

e.g.

$ pkcon refresh
Refreshing cache                        [=========================]         
Starting                                [=========================]         
Refreshing software list                [=========================]         
Finished                                [=========================]         
$ pkcon resolve firefox
Resolving                               [=========================]         
Fatal error: Package firefox not found

or

pkcon install some-package
pkcon remove some-package

You have to wait some seconds for the backend to become available again. Using --cache-age 1 might resolve the issue.

[livingsilver94]

Maybe there's a lock being released too early, or not created at all?

[livingsilver94]

Instead of the custom eopkgBackend.py, I converted the upstream pisiBackend.py to Python3. Using that backend the error is more explicit (probably the exception isn't being catched, which is a good thing in this case).

I ran: pkcon remove micro; pkcon remove micro and that's the error:

Command failed: This tool could not find the installed package: Error Type: <class 'pisi.pxml.xmlfile.Error'>
Error Value: Unable to read file (/var/lib/eopkg/package/micro-2.0.13-29/metadata.xml): Error reading file '/var/lib/eopkg/package/micro-2.0.13-29/metadata.xml': failed to load external entity "/var/lib/eopkg/package/micro-2.0.13-29/metadata.xml"
  File : /usr/share/PackageKit/helpers/eopkg/eopkgBackend.py, line 593, in <module>
    main()
  File : /usr/share/PackageKit/helpers/eopkg/eopkgBackend.py, line 590, in main
    backend.dispatcher(sys.argv[1:])
  File : /usr/lib/python3.11/site-packages/packagekit/backend.py, line 740, in dispatcher
    self.dispatch_command(args[0], args[1:])
  File : /usr/lib/python3.11/site-packages/packagekit/backend.py, line 672, in dispatch_command
    self.resolve(filters, package_ids)
  File : /usr/share/PackageKit/helpers/eopkg/eopkgBackend.py, line 483, in resolve
    self.__get_package(package[0], filters)
  File : /usr/share/PackageKit/helpers/eopkg/eopkgBackend.py, line 112, in __get_package
    pkg = self.installdb.get_package(package)
  File : /usr/lib/python3.11/site-packages/pisi/db/installdb.py, line 262, in get_package
    metadata.read(metadata_xml)
  File : /usr/lib/python3.11/site-packages/pisi/pxml/autoxml.py, line 480, in read
    read_xml = self.readxml(
  File : /usr/lib/python3.11/site-packages/pisi/pxml/xmlfile.py, line 86, in readxml
    raise Error(_("Unable to read file ($s): $s") $ (localpath, e))

I also tried to remove another package, tealdeer in my case. Same error in the metadata file.

Could also well be that that is a different error...

[livingsilver94]

No, I confirm it's the same error, just captured by this line: https://github.com/getsolus/PackageKit/blob/eopkg4-bin/backends/eopkg/eopkgBackend.py#L823

[joebonrichie]

is it as simple as __get_package() / resolve() needing the @privileged decorator?

[livingsilver94]

I sincerely don't know but it's so easy it's worth a try. I'll be able to test it tomorrow though, feel free to go ahead if you can.

[livingsilver94]

Nope, doesn't work.

[livingsilver94]

I can't for the life of me find why the InstallDB isn't updated until a few seconds passed. The pisi backend is marked to be unparallel.

[sheepman4267]

I can't for the life of me find why the InstallDB isn't updated until a few seconds passed. The pisi backend is marked to be unparallel.

Turns out, the DB was being updated correctly, and nothing was running in parallel. However, our package/component/file/install/etc. DB objects were being invalidated by the PiSi API.

This issue is fixed (for me) by bee531c. More details in that commit's message.

[livingsilver94]

First off, congratulations! I was having a hard time finding motivation to work on eopkg code. Good job, really.

Are you sure nothing's parallel? I was able to bypass the error by waiting that packagekitd closes the running child. I could ensure that by reading its log. Anyway, just curiosity.

[sheepman4267]

@livingsilver94 Yes, I'm pretty sure nothing is parallel. Packagekit's spawn backend system seems to leave the child process running and communicate with it to run subsequent jobs, rather than kill it after every job. In other words, if you've disabled parallel operation, PackageKit waits to get the finished signal before requesting the next job from the same backend process rather than always re-executing the whole backend.

I get the idea here. It makes things faster by not repeating all the setup boilerplate for each PK action. The fact that we're writing a backend that talks to a horrible API which basically shoots itself in the foot every time you ask it to do anything is not PackageKit's fault.

A task for later is to figure out why all @locked methods of the PiSi API need to invalidate all currently-open database objects, which might have a speed impact on eopkg.... but you won't find me doing that until we have all the features working.