TornadoPilotLogging is too resource intensive

[chrisburr]

The TornadoPilotLogging is completely overloading Tornado in certification despite their being very few jobs. Even editing the code to prevent it touching the file system wasn't enough to improve the situation.

Calling Client(...).ping() was taking 3-10 seconds with it enabled. Disabling the pilot logging reduced the ping to ~70ms.

I edited the certification CS to disable it using:

         TornadoPilotLogging
         {
-          Protocol = https
+          # Protocol = https

[martynia]

I disabled remote pilot logging (RemoteLogging=False) for both VOs. This would only affect new pilot submissions, so we have to wait a bit and then try to enable the remote logging taking into account considerations below. I think there are 2 possible actions to reduce resource intensity:

1. Make the buffer size configurable. It is 10 lines at the moment by default but there is no flag to modify it. In a short run one could set it to 1000 lines, say, they there would be practically one remote call per command + the initial buffer. This is easy to do:
   https://github.com/DIRACGrid/Pilot/blob/a50a6b636e4a8d51fea1adfd5f2b97d99b95ad3a/Pilot/pilotTools.py#L342
   

by adding the buffsize=<value from CS> or even hot fix the default value here:

https://github.com/DIRACGrid/Pilot/blob/a50a6b636e4a8d51fea1adfd5f2b97d99b95ad3a/Pilot/pilotTools.py#L388

2. Make fine resource selection: site or even CE when the remote logger would be enabled. Enabling on a VO by VO basis is apparently too heavy. This requires some coding and adding new options to the CS.

cheers, Janusz

[fstagni]

Option 2 might be a welcome addition, but I consider it at the moment like a feature request. The real question here is: what is the reason for the overloading? What's using the resources?

[martynia]

Hi all, I’m assuming (if I understand Chris B correctly) that Chris commented out log writing at the Tornado end and there was no improvement. It must be network then. In default setting we have at the moment, a message is sent to Tornado every 10 lines of logs, There are exceptions from this rule, one being the end of a command when a remaining buffer is sent, but I think a length of 10 lines is a fair assumption. A typical, finalised log file size is ~200kB, one I looked at has 2558 lines, which means there are ~ 260 calls to Tornado per pilot. By enlarging a buffer we can reduce the number of calls by 2 orders of magnitude. I’v just seen a complaint from RAL about 25k jobs sent there. Just for RAL. Is it the load Tornado can cope with ? I think the VO scope is too wide. We can look at this tomorrow during the meeting, maybe. Cheers, JM

…

On 22 Jan 2024, at 08:29, Federico Stagni ***@***.***> wrote: Option 2 might be a welcome addition, but I consider it at the moment like a feature request. The real question here is: what is the reason for the overloading? What's using the resources? — Reply to this email directly, view it on GitHub <#7411 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB2KE7OGG4QVSC3ZARQWP6DYPYIM7AVCNFSM6AAAAABCD3LY7KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBTGQYDINZYHE>. You are receiving this because you were assigned.

[fstagni]

Let's mark it as solved.

[martynia]

OK, there is still a pending pilot PR #230/ You merged the Dirac counterpart already.