buildRequestCollapsing not working in 0.9.6 as expected

[sphet]

I am trying to get build request merging to work in 0.9 - we are using it effectively in 0.8.6. The documentation states:

Sourcestamps are compatible if all of the below conditions are met:

Their codebase, branch, project, and repository attributes match exactly
Neither source stamp has a patch (e.g., from a try scheduler)
Either both source stamps are associated with changes, or neither are associated with changes but they have matching revisions.

In my case I have a long-running builder - it takes five minutes to complete - and I submit the following sequence of changes to Perforce:

edit & submit fileA.txt at 10:00AM ; CL461078 --> builder starts
edit & submit fileB.txt at 10:02AM ; CL461079 --> builder is busy, submission is pending
edit & submit fileC.txt at 10:03AM ; CL461080 --> builder is busy, submission is pending

Once CL001 is finished building, I would expect CL002 and CL003 to be collapsed into one build request. This is not happening. I have configued the builder with 'collapseRequests=True'. The debug output shown for the source stamps is:

[{"branch":null,"codebase":"","created_at":1493227979,"patch":null,"project":"","repository":"","revision":"461078","ssid":427}]
[{"branch":null,"codebase":"","created_at":1493228009,"patch":null,"project":"","repository":"","revision":"461079","ssid":428}]
[{"branch":null,"codebase":"","created_at":1493228039,"patch":null,"project":"","repository":"","revision":"461080","ssid":429}]

Looking at the source code I don't really see any code that would handle the 'both source stamps are associated with changes', as certainly in the canBeCollapsed function in buildRequest.pyL254 the collapse is failing because the revisions are different.

Is my reading of the documentation wrong? In 8.6 it seemed this worked - stacked up pending requests (regardless of revision number) would all be built once the builder is free, not atomically. How would we modify the code to handle this?

Reviewing the source code to 0.8.x I see that in sourcestamp.py the canBeMergedWith reads, in addition to other things:

    if self.changes and other.changes:
        return True
    elif self.changes and not other.changes:
        return False  # we're using changes, they aren't
    elif not self.changes and other.changes:
        return False  # they're using changes, we aren't

I don't see parallel code in the 0.9 branch.

[tardyp]

Indeed, problem comes from this very old change from the begining of nine refactor:
a3fcc91#diff-e28bcb3c5230c2d392829647506762daL169

The new algorithm does not implement the condition documented: "Either both source stamps are associated with changes"

When looking at the coverage for this code, it looks like there is no unit test for it.

[sphet]

I am working through the code now to see if I sort it out, but it seems the changes are associated with a builder's source, and not sourcestamps. Given the two build requests passed to canBeCollapsed, I think I can query the sources and go from there. I'll give it a try.

[noc0lour]

For testing you could write a unit test and put prints into the source, so you don't have to run a test-master. That's how I test changes like this often ;)
Just install buildbot in a virtualenv with pip install -e master

[sphet]

I am not sure if my patch is sufficiently safeguarded, but I ran it in a couple of test environments and it seems to work. I do not know the code base enough - perhaps a moderator can look it over..

[vigsterkr]

any update on this one as i'm experiencing the same problem on 0.9.8

[sphet]

Sorry I ran out of steam on this since I don't have a good setup for writing the unit tests. I ended up taking the logic here and putting it into a client-side 'merge request' callback method for now in my master.cfg. I essentially duplicated the code from my patch and stuck it into a function there. I hope to come back around to this since I would like it in the mainline for maintainability, but I don't know enough about writing unit tests to quickly get one going.

…

On Mon, Jun 26, 2017 at 7:58 PM, Viktor Gal ***@***.***> wrote: any update on this one as i'm experiencing the same problem on 0.9.8 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3151 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACnpqCdd_YQ5PCFZHFcuoGTG_QtdGiFFks5sIG_ogaJpZM4NJP78> .

[tardyp]

@sphet what do you mean by a good setup for writing the unit tests?

did you see my answer here?
#3152 (comment)

what could I have done better to help you?

[sphet]

It's not you it's me. I am not familiar enough with the tools of open source deployment (git, pip, virtualenv) etc. I'll try to find the time but I am old dog - learning new tricks is exhausting. :-)

…

On Wed, Jun 28, 2017 at 4:07 AM Pierre Tardy ***@***.***> wrote: @sphet <https://github.com/sphet> what do you mean by a good setup for writing the unit tests? did you see my answer here? #3152 (comment) <#3152 (comment)> what could I have done better to help you? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3151 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACnpqPWGO2Tzg87pK6zhdMJaGWxRhRRUks5sIjQBgaJpZM4NJP78> .