Implements virtually accept the delete request #1136

nilxam · 2017-09-12T11:40:00Z

https://progress.opensuse.org/issues/20420

Workflow:

selecting a request to a staging, the pseudometa is changed to
{id: 100, package: FOO, author: user, type: submit}
{id: 101, package: BAR, author: user, type: delete}
during accepting a delete request:
If 'type' is exists in pseudometa and delreq-monitor was set in conf.py, the delete request will through virtual accept process, the binary will be removed on standard repo. If 'type' is not exists, go through the old route make sure it will not break the current stuff.
in the later check-in round:
If it has found the virtual accepted request's package doesn't have binary on any build repo in the main project, the accept command accepts delreq-monitor review and continue accepts this request to the main project via accept_other_new().

coveralls · 2017-09-12T12:10:09Z

Coverage decreased (-0.4%) to 47.654% when pulling 3ce8ccc on nilxam:virtual_accept into 3c1a264 on openSUSE:master.

nilxam · 2017-09-12T12:38:10Z

TODO:

don't need delreq-monitor anymore, replace it to factory-maintainers group. [done]
verify sub-package's binary also empty. [done]
remove the sub-package before merge the request. [done]

coveralls · 2017-09-13T10:17:31Z

Coverage decreased (-0.5%) to 47.586% when pulling 294da92 on nilxam:virtual_accept into 3c1a264 on openSUSE:master.

nilxam · 2017-09-13T10:40:46Z

TODO list is clean up, please review!

coveralls · 2017-09-13T10:56:27Z

Coverage decreased (-0.6%) to 47.479% when pulling dbb7340 on nilxam:virtual_accept into 3c1a264 on openSUSE:master.

coveralls · 2017-09-13T11:13:25Z

Coverage decreased (-0.6%) to 47.479% when pulling 62d5f7d on nilxam:virtual_accept into 3c1a264 on openSUSE:master.

coveralls · 2017-09-13T11:56:21Z

Coverage decreased (-0.6%) to 47.479% when pulling 62d5f7d on nilxam:virtual_accept into 3c1a264 on openSUSE:master.

jberry-suse

Overall, seems like a nice approach to the problems faced with delete requests. A few areas that would benefit from refactoring and some minor wording changes.

Although the 8 commits have nice messages it would appear 4 fall in the category of fixup as git would call them and should be squashed into the commits that originally added the code.

accept_command: change delreq-monitor review to the maintainer group
accept_command: ensure sub-package's binary are empty
accept_command: accepting delreq-monitor group review first
accept_command: make sure project's pseudometa have the type attribute

As noted in a review comment I assume the following commit is a bonus feature implemented in this request as otherwise I would expect it to be desirable for the non-virtual handling? No change, just curious.

accept_command: delete sub-package links before accepting the delete request

jberry-suse · 2017-09-14T03:14:14Z

osclib/accept_command.py

+        pkgs = []
+        filelist = self.api.get_filelist_for_package(pkgname=package, project=self.api.project, expand='1', extension='spec')
+        for spec in filelist:
+            pkgs.append(spec[:-5])


Since this same sort of logic is done in #1110, perhaps extracting as another method in StagingAPI and using for both would make sense (ie a method to return list of subpackages based on spec files in main package). Might as well merge the other and refactor in this request along with all the additional usages.

jberry-suse · 2017-09-14T03:20:05Z

osclib/accept_command.py

+
+        for pkg in pkgs:
+            query = {'view': 'binarylist', 'package': pkg, 'multibuild': '1'}
+            pkg_binarylist = ET.parse(http_GET(self.api.makeurl(['build', project, '_result'], query=query))).getroot()


Without running the queries to double check this seems like the same information provided by osclib.core.package_binary_list(). From the comment and my memory multibuild results are returned there as well. Perhaps try it and just and check if that returns an non-empty list. Otherwise the binary_list() method is the one used by duplicate binary check.

The main difference is the details of the information returned, but you only seem to care if any binaries or not so I would expect them to be sufficient.

hmm... I don't want to hardcoding arch, the binary must be removed in any arch.

jberry-suse · 2017-09-14T03:23:50Z

osclib/accept_command.py

+        pkgs = []
+        filelist = self.api.get_filelist_for_package(pkgname=package, project=self.api.project, expand='1', extension='spec')
+        for spec in filelist:
+            pkgs.append(spec[:-5])


Another need for having a method to return list of subpackages based on spec files as noted above.

jberry-suse · 2017-09-14T03:24:38Z

osclib/accept_command.py

+        for pkg in pkgs:
+            meta = show_package_meta(self.api.apiurl, self.api.project, pkg)
+            meta = ''.join(meta)
+            # Update package meta with disabling build


nit: s/with disabling/to disable/

jberry-suse · 2017-09-14T03:25:37Z

osclib/accept_command.py

+            meta = ''.join(meta)
+            # Update package meta with disabling build
+            self.api.create_package_container(self.api.project, pkg, meta=meta, disable_build=True)
+            url = self.api.makeurl(['build', self.api.project], {'cmd': 'wipe', 'repository': self.api.main_repo, 'package': pkg})


osc.core.wipebinaries()

jberry-suse · 2017-09-14T03:36:10Z

osclib/conf.py

@@ -44,6 +44,8 @@
        'openqa': 'https://openqa.opensuse.org',
        'lock': 'openSUSE:%(project)s:Staging',
        'lock-ns': 'openSUSE',
+        'delreq-monitor': 'factory-maintainers',
+        'main-repo': 'standard',


This should likely be used in repo_checker as well since for lack of having a proper variable I kept the hard-coded standard. I'll update that once this is merged.

jberry-suse · 2017-09-14T03:38:52Z

osclib/conf.py

@@ -44,6 +44,8 @@
        'openqa': 'https://openqa.opensuse.org',
        'lock': 'openSUSE:%(project)s:Staging',
        'lock-ns': 'openSUSE',
+        'delreq-monitor': 'factory-maintainers',


I am not clear what is the purpose behind the word monitor in the key. Seems more like a virtual delete request queue or backlog or pending list. I understand the list will be monitored to determine when it is appropriate to actually accept the delete, but given this controls the virtual delete request feature it seems like that would be more appropriate to be in the key name.

Perhaps virtual-delete-review or somesuch as that provides a much better idea of what this is for at a glance.

jberry-suse · 2017-09-14T03:41:26Z

osclib/stagingapi.py

@@ -1254,7 +1263,7 @@ def list_requests_in_prj(self, project):

        return list

-    def add_review(self, request_id, by_project=None, by_group=None, msg=None):
+    def add_review(self, request_id, by_project=None, by_group=None, by_user=None, msg=None):


I assume this is no longer necessary since using factory-maintainers group?

jberry-suse · 2017-09-14T03:42:20Z

osclib/conf.py

@@ -58,6 +60,8 @@
        'openqa': 'https://openqa.opensuse.org',
        'lock': 'openSUSE:%(project)s:Staging',
        'lock-ns': 'openSUSE',
+        'delreq-monitor': 'factory-maintainers',


Would it make sense to use a different user/group like you originally did to avoid the continued mixing of factory in non-factory projects?

the idea here was that 'the one that will have to accept it into oS:F in the end must be in factory-maintainers' anyway; and the 'last step', removing the review of the 'delreq-monitor', would happen directly in the osc staging accept call; so definitively by a user in the factory-maintainers group.

If we take a 2nd group, it must have all the member of factory-maintainers, plus potentially more (which did not seem to realistic)

So I guess this is a symptom of the larger problem that that group is also used for Leap already. As such it makes sense to use in this context, but should be considered for changing for Leap in general at some point.

indeed - so fo far the group allowed to write into the product directly was small and the same set of users is used across TW and Leap (to back each other up);

So perhaps masters-of-the-obs :)

That sounds more like the obs admins; happy not to be part of that (most of the times)

jberry-suse · 2017-09-14T03:44:48Z

osclib/accept_command.py

+
+        return True
+
+    def find_virtual_accepted_requests(self, project):


Rather than duplicate all the dom walking code from find_new_requests() it would seem to make more sense to rename that method to somethign like find_requests() and add a parameter to indicate looking for virtual delete requests.

coveralls · 2017-09-26T11:11:23Z

Coverage decreased (-0.4%) to 47.531% when pulling 1f0d075 on nilxam:virtual_accept into 4011159 on openSUSE:master.

nilxam · 2017-09-26T11:12:47Z

updated, still need to test it in real.

coveralls · 2017-09-26T12:29:40Z

Coverage decreased (-0.5%) to 47.437% when pulling a34901d on nilxam:virtual_accept into 4011159 on openSUSE:master.

coveralls · 2017-09-26T13:59:45Z

Coverage decreased (-0.5%) to 47.464% when pulling bbe833b on nilxam:virtual_accept into 4011159 on openSUSE:master.

DimStar77 · 2017-09-27T14:50:31Z

Fai

DELETE https://api.opensuse.org/source/openSUSE:Factory:Staging:D:DVD/ecj-bootstrap?comment=ready+to+accept&force=1
GET https://api.opensuse.org/request/527833?withfullhistory=1
GET https://api.opensuse.org/request/527833?withfullhistory=1
GET https://api.opensuse.org/request/527833?withfullhistory=1
POST https://api.opensuse.org/request/527833?newstate=accepted&cmd=changereviewstate&by_project=openSUSE%3AFactory%3AStaging%3AD
Accepting staging review for ecj-bootstrap
CACHE_MISS https://api.opensuse.org/source/openSUSE:Factory/ecj-bootstrap?extension=spec (does not exist)
GET https://api.opensuse.org/source/openSUSE:Factory/ecj-bootstrap?extension=spec
CACHE_PUT https://api.opensuse.org/source/openSUSE:Factory/ecj-bootstrap?extension=spec openSUSE:Factory
GET https://api.opensuse.org/source/openSUSE:Factory:Staging/_attribute/openSUSE:LockedBy
POST https://api.opensuse.org/source/openSUSE:Factory:Staging/_attribute/openSUSE:LockedBy
Traceback (most recent call last):
  File "/usr/bin/osc", line 41, in <module>
    r = babysitter.run(osccli)
  File "/usr/lib/python2.7/site-packages/osc/babysitter.py", line 61, in run
    return prg.main(argv)
  File "/usr/lib/python2.7/site-packages/osc/cmdln.py", line 344, in main
    return self.cmd(args)
  File "/usr/lib/python2.7/site-packages/osc/cmdln.py", line 367, in cmd
    retval = self.onecmd(argv)
  File "/usr/lib/python2.7/site-packages/osc/cmdln.py", line 501, in onecmd
    return self._dispatch_cmd(handler, argv)
  File "/usr/lib/python2.7/site-packages/osc/cmdln.py", line 1232, in _dispatch_cmd
    return handler(argv[0], opts, *args)
  File "/home/dimstar_suse/.osc-plugins/osc-staging.py", line 433, in do_staging
    if cmd.perform(api.prj_from_letter(prj), opts.force):
  File "/home/dimstar_suse/.osc-plugins/osclib/accept_command.py", line 130, in perform
    self.virtually_accept_delete(req['id'], req['package'])
  File "/home/dimstar_suse/.osc-plugins/osclib/accept_command.py", line 88, in virtually_accept_delete
    self.api.add_review(request_id, by_group=self.api.delreq_reivew)
AttributeError: 'StagingAPI' object has no attribute 'delreq_reivew'

DimStar77 · 2017-09-27T14:55:52Z

osclib/accept_command.py

@@ -51,6 +84,20 @@ def reset_rebuild_data(self, project):
        content = ET.tostring(root)
        http_PUT(url + '?comment=accept+command+update', data=content)

+    def virtually_accept_delete(self, request_id, package):
+        self.api.add_review(request_id, by_group=self.api.delreq_reivew)


typo.. this should be self.api.delreq_review

jberry-suse · 2017-09-27T22:04:48Z

osclib/accept_command.py

            change_request_state(self.api.apiurl, str(req['id']), 'accepted', message='Accept to %s' % self.api.project)
            # Check if all .spec files of the package we just accepted has a package container to build
            self.create_new_links(self.api.project, req['packages'][0], oldspecs)
            changed = True

        return changed

+    def remove_obsoleted_develtag(self, project, package):
+        xpath = {
+            'package': "@project='%s' and devel/@project='%s' and devel/@package='%s'" % (project, project, package),


This removes the sub-packages which have their devel project set to the target project right? Not sure if you like, but one tweak that I checked works is:

@project='%s' and devel/@project=@project and devel/@package='%s'

to avoid adding project twice.

No, it only removes the devel attribute from the packages - it does not remove the package (but as long as the devel attribute is set, accepting the delete request fails; with or without the virtual delete

It only removes the devel tag from the package if it has set devel pacakge in the same project.

Yes, that's what I meant. Obviously, deleting a devel project would be a bit crazy.

jberry-suse · 2017-09-27T22:07:32Z

osclib/accept_command.py

@@ -14,7 +14,40 @@ def __init__(self, api):
        self.api = api

    def find_new_requests(self, project):
-        query = "match=state/@name='new'+and+(action/target/@project='{}'+and+action/@type='submit')".format(project)
+        query = "match=state/@name='new'+and+action/target/@project='{}'".format(project)


This will now include things like change_devel requests. Are you sure the rest of the code can handle them properly or should it explicitly look for submit or delete requests?

I had to restricted explicitly for SLE in #1121 which is the equivalent of change_devel in Factory so it be good to make sure we do not introduce another case.

any kind of request must be reviewed by human and bot, if it's state is new then means all reviews are passed, in the past, it was merged manually(either change_devel request to Factory or add_role to SLE), this change just auto-merge those kind of request while executing accept_other_new.

I understand they were reviewed, a question of can the code handle the structure of those different types of requests or does it make assumptions as the code I had to restrict did.

jberry-suse · 2017-09-27T22:08:57Z

osclib/accept_command.py

+        }
+        collection = search(self.api.apiurl, **xpath)['package']
+        pkglist = [p.attrib['name'] for p in collection.findall('package')]
+        for pkg in pkglist:


Any reason to build a list and then iterate over it rather than just looping over the original data structure?

for pkg in collection.findall('package'): set_devel_project(self.api.apiurl, project, pkg.attrib['name'], devprj=None)

just different coding style not you like...

Indeed although a bit more than just style given the performance impact albeit tiny, nothing incorrect about it, just curious as typically looping twice is worse than looping once.

Well, loops an extra time and allocates an entire extra array. Again tiny, but I see no upside. From readability standpoint I now have to understand the transformation from original structure to new one and keep the new on in my head while reading the loop instead of just one structure, no transformation, and access data directly the same way it is done countless other places.

jberry-suse · 2017-09-27T22:10:52Z

osclib/accept_command.py

-                                 message='Accept to %s' % self.api.project)
-            self.create_new_links(self.api.project, req['package'], oldspecs)
+            if 'type' in req and req['type'] == 'delete' and self.api.delreq_review:
+                # Virtually accept to the delete request


nit: s/to //

Although really it now says almost exactly the same thing the method name does which isn't terribly helpful.

coveralls · 2017-09-28T08:34:59Z

Coverage decreased (-0.5%) to 47.464% when pulling 30eb46f on nilxam:virtual_accept into 4011159 on openSUSE:master.

DimStar77 · 2017-09-28T10:49:24Z

Last accept round was done with this PR merged on my local branch

The package python-argparse was handled by the delete process as defined:

the package has been build disabled in openSUSE:Factory
The binaries have been removed from openSUSE:Factory/standard (but kept in /totest and /snapshot)
The actual delete request recevied a new review for 'factory-maintainers'; no more in the staging backlog

Once the current snapshot (0928) is being published, the binaries should all be gone and the 'accepting of the delete request' should be completed (to be seen)

coveralls · 2017-09-28T11:56:38Z

Coverage decreased (-0.5%) to 47.464% when pulling 27ba1fe on nilxam:virtual_accept into 3bbeacd on openSUSE:master.

jberry-suse

Assuming the drop of request type filter in find_new_requests() does not cause issues with change_devel requests. It would seem to make sense to restrict to the types that the code actually wants to handle.

Thanks for improving the code, and overall seems like a nice solution.

coveralls · 2017-09-29T10:23:01Z

Coverage decreased (-0.5%) to 47.437% when pulling f5e7c2e on nilxam:virtual_accept into 3bbeacd on openSUSE:master.

DimStar77 · 2017-09-29T10:27:27Z

Assuming the drop of request type filter in find_new_requests() does not cause issues with change_devel requests. It would seem to make sense to restrict to the types that the code actually wants to handle.

Imho, accept_other_new should really handle all 'new' requests; any incoming request on openSUSE:Factory receives a 'review' at least on factory-staging and a couple bots.

at least factory-staging is a 'human review' for non-submit and non-delete requests, as those are not grouped into staging projects (and staging list / staging select ignores them)

nilxam · 2017-09-29T13:14:21Z

Added an additional change to cope with 'virtual accept' in Rings. The package is deleted in theory thus should happened in Rings for real.

if self.api.ring_packages.get(pkg):
    delete_package(self.api.apiurl, self.api.ring_packages.get(pkg), pkg, force=True, msg="Cleanup package in Rings")

coveralls · 2017-09-29T13:42:52Z

Coverage decreased (-0.5%) to 47.41% when pulling 61295d0 on nilxam:virtual_accept into 3bbeacd on openSUSE:master.

Implementation virtually accept the delete request, the delete request will be added another delreq-review review and also wipes the binary in the main project, the backend will sync 'nothing' to ToTest and Snapshot after all. Once all repository has no binary then remove the package in real.

coveralls · 2017-10-02T09:22:19Z

Coverage decreased (-0.5%) to 47.41% when pulling 35679d1 on nilxam:virtual_accept into 3bbeacd on openSUSE:master.

jberry-suse suggested changes Sep 14, 2017

View reviewed changes

nilxam force-pushed the virtual_accept branch from 62d5f7d to 1f0d075 Compare September 26, 2017 11:08

nilxam changed the title ~~[DO NOT MERGE] Implements virtual accept to the delete request~~ [DO NOT MERGE] Implements virtually accept the delete request Sep 26, 2017

nilxam force-pushed the virtual_accept branch from a34901d to bbe833b Compare September 26, 2017 13:44

DimStar77 reviewed Sep 27, 2017

View reviewed changes

jberry-suse reviewed Sep 27, 2017

View reviewed changes

nilxam force-pushed the virtual_accept branch from bbe833b to 30eb46f Compare September 28, 2017 08:31

nilxam force-pushed the virtual_accept branch from 30eb46f to 27ba1fe Compare September 28, 2017 11:51

jberry-suse approved these changes Sep 28, 2017

View reviewed changes

nilxam force-pushed the virtual_accept branch from 27ba1fe to f5e7c2e Compare September 29, 2017 10:19

nilxam force-pushed the virtual_accept branch from f5e7c2e to 61295d0 Compare September 29, 2017 13:10

nilxam changed the title ~~[DO NOT MERGE] Implements virtually accept the delete request~~ Implements virtually accept the delete request Oct 2, 2017

nilxam added 2 commits October 2, 2017 17:15

tests: fix testsuite for virtually accept

35679d1

nilxam force-pushed the virtual_accept branch from 61295d0 to 35679d1 Compare October 2, 2017 09:18

nilxam merged commit a306181 into openSUSE:master Oct 2, 2017

nilxam deleted the virtual_accept branch November 15, 2019 09:55


		return True

		def find_virtual_accepted_requests(self, project):

Implements virtually accept the delete request #1136

Implements virtually accept the delete request #1136

Conversation

nilxam commented Sep 12, 2017 • edited

coveralls commented Sep 12, 2017

nilxam commented Sep 12, 2017 • edited

coveralls commented Sep 13, 2017

nilxam commented Sep 13, 2017

coveralls commented Sep 13, 2017

coveralls commented Sep 13, 2017

coveralls commented Sep 13, 2017

jberry-suse left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Sep 26, 2017

nilxam commented Sep 26, 2017

coveralls commented Sep 26, 2017

coveralls commented Sep 26, 2017

DimStar77 commented Sep 27, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jberry-suse Sep 27, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jberry-suse Sep 28, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jberry-suse Sep 28, 2017 • edited

Choose a reason for hiding this comment

jberry-suse Sep 28, 2017 • edited

Choose a reason for hiding this comment

jberry-suse Sep 27, 2017 • edited

Choose a reason for hiding this comment

coveralls commented Sep 28, 2017

DimStar77 commented Sep 28, 2017

coveralls commented Sep 28, 2017

jberry-suse left a comment • edited

Choose a reason for hiding this comment

coveralls commented Sep 29, 2017

DimStar77 commented Sep 29, 2017

nilxam commented Sep 29, 2017

coveralls commented Sep 29, 2017

coveralls commented Oct 2, 2017

nilxam commented Sep 12, 2017 •

edited

nilxam commented Sep 12, 2017 •

edited

jberry-suse Sep 27, 2017 •

edited

jberry-suse Sep 28, 2017 •

edited

jberry-suse Sep 28, 2017 •

edited

jberry-suse Sep 28, 2017 •

edited

jberry-suse Sep 27, 2017 •

edited

jberry-suse left a comment •

edited