Creating submit requests on large packages take lot of time

[scarabeusiv]

When working on packages like chromium and libreoffice the OBS times out quite often and even if the submit operation is successful it takes quite a lot of time:

time osc sr network:chromium/chromium-beta network:chromium/chromium -m "up"
created request id 792360

real	8m59,322s
user	0m0,295s
sys	0m0,044s

osc sr network:chromium/chromium-beta network:chromium/chromium -m "up"
Server returned an error: HTTP Error 502: Bad Gateway

Request: https://api.opensuse.org/request?cmd=create
Headers:
Date: Wed, 08 Apr 2020 08:11:53 GMT
Server: Apache
Vary: accept-language,accept-charset
Strict-Transport-Security: max-age=31536000
Upgrade: h2
Connection: Upgrade, close
Accept-Ranges: bytes
Transfer-Encoding: chunked
Content-Type: text/html; charset=utf-8
Content-Language: en

If the OBS errors out some time later I actually get the request created see 792361 792360 792359 but it is really weird.

Could it be possible to somehow speed up the operations on these big packages in some way? I dunno maybe it does tarball diffing that could be ommited/etc...

[scarabeusiv]

This is getting worse and worse, I manage to create any operations on LO/chromium on every nth attempt wasting hours waiting for OBS to do something on background.

[coolo]

https://obs-measure.opensuse.org/d/fzuOfHQik/request?orgId=1&var-request_id=d8bec119-5eb7-4c57-8b73-d5db166c9801&from=1594972877872&to=1594973868644&var-method=RequestController%23global_command reports it as 17 minutes.

I, [2020-07-17T08:01:17.873488 #10471]  INFO -- : [d8bec119-5eb7-4c57-8b73-d5db166c9801] [10471:80004.36] iChain user extracted from header: scarabeus_iv
I, [2020-07-17T08:17:48.645379 #10471]  INFO -- : [d8bec119-5eb7-4c57-8b73-d5db166c9801] [10471:80995.13] method=POST path=/request format=xml controller=RequestController action=global_command status=400 duration=990772.55 view=1.52 db=59.05 params={"cmd"=>"create"} host=192.168.2.131 time=1296594.15 backend=990578.86 user=scarabeus_iv

Basically all of this time it's this call into the backend:
POST source/network:chromium/chromium?cmd=diff&expand=1&filelimit=10000&opackage=chromium.12991&oproject=openSUSE%3ALeap%3A15.2%3AUpdate&rev=891f50c314cdc568c986a5d9cca0cbf9&tarlimit=10000

which took 990s

[coolo]

The backtrace is this:

[ce6a9b6a-ed43-44e2-9315-efeb6d0f481f] app/models/bs_request_action/differ/for_source.rb:12:in `perform'
[ce6a9b6a-ed43-44e2-9315-efeb6d0f481f] app/models/bs_request_action/differ.rb:12:in `sourcediff'
[ce6a9b6a-ed43-44e2-9315-efeb6d0f481f] app/models/bs_request_action.rb:245:in `contains_change?'
[ce6a9b6a-ed43-44e2-9315-efeb6d0f481f] app/models/bs_request_action.rb:686:in `expand_targets'
[ce6a9b6a-ed43-44e2-9315-efeb6d0f481f] app/models/bs_request.rb:1061:in `block in expand_targets'
[ce6a9b6a-ed43-44e2-9315-efeb6d0f481f] app/models/bs_request.rb:1060:in `expand_targets'
[ce6a9b6a-ed43-44e2-9315-efeb6d0f481f] app/models/bs_request.rb:940:in `sanitize!'
[ce6a9b6a-ed43-44e2-9315-efeb6d0f481f] app/controllers/request_controller.rb:151:in `block in request_create'
[ce6a9b6a-ed43-44e2-9315-efeb6d0f481f] app/controllers/request_controller.rb:145:in `request_create'
[ce6a9b6a-ed43-44e2-9315-efeb6d0f481f] app/controllers/request_controller.rb:69:in `global_command'
[ce6a9b6a-ed43-44e2-9315-efeb6d0f481f] config/initializers/wrap_parameters.rb:38:in `call'

So no background job, it's trying to figure if there are changes at all to avoid creating actions without source change (basically since 98ff007). Perhaps there is a cheaper way to catch that specific problem? E.g. compare expanded srcmd5?