Forwarder eventually stops sending logs

[chuseman]

We've tested the new forwarder functionality to replace the old seqfwd app. After running correctly for a few days, we've observed several instances of this message in logs across several boxes (all running as a service on Windows):

{
  "@t":"2026-03-05T20:45:30.6719054Z",
  "@m":"Forwarding log shipper failed and exited",
  "@i":"1efb7888",
  "@l":"Fatal",
  "@x":"System.UnauthorizedAccessException: Access to the path 'C:\\ProgramData\\Seqcli\\Forwarder\\SeqCli\\Buffer\\fd5a6b94b1bf49df954745f8262a5c2d\\0000000000000001.clef' is denied.\r\n   at System.IO.FileSystem.DeleteFile(String fullPath)\r\n   at SeqCli.Forwarder.Filesystem.System.SystemStoreDirectory.TryDelete(String name) in D:\\a\\seqcli\\seqcli\\src\\SeqCli\\Forwarder\\Filesystem\\System\\SystemStoreDirectory.cs:line 96\r\n   at SeqCli.Forwarder.Storage.BufferReader.AdvanceTo(BufferPosition newReaderHead) in D:\\a\\seqcli\\seqcli\\src\\SeqCli\\Forwarder\\Storage\\BufferReader.cs:line 248\r\n   at SeqCli.Forwarder.Channel.ForwardingChannel.<>c__DisplayClass5_0.<<-ctor>b__1>d.MoveNext() in D:\\a\\seqcli\\seqcli\\src\\SeqCli\\Forwarder\\Channel\\ForwardingChannel.cs:line 115",
  "@tr":"39d89a8ca8d77ce084232c6be16c588b",
  "@sp":"8e85c1e1a8c13011",
  "SourceContext":"SeqCli.Forwarder.Channel.ForwardingChannel",
  "RequestId":"0HNJQ5TSLESBP:00000001",
  "RequestPath":"/api/events/raw",
  "ConnectionId":"0HNJQ5TSLESBP",
  "MachineName":"xxx",
  "Application":"SeqCli Forwarder"
}

Our configuration is pretty simple - the only interesting bit is that we've set useApiKeyForwarding to true in the json:

{
  "connection": {
    "serverUrl": "https://xxx",
    "apiKey": null,
    "pooledConnectionLifetimeMilliseconds": null,
    "eventSizeLimitBytes": 262144,
    "batchSizeLimitBytes": 10485760
  },
  "output": {
    "disableColor": false,
    "forceColor": false
  },
  "forwarder": {
    "storage": {
      "targetChunkSizeBytes": 52428800,
      "maxChunks": null
    },
    "diagnostics": {
      "internalLoggingLevel": "Information",
      "internalLogServerUri": null,
      "internalLogServerApiKey": null,
      "exposeIngestionLog": false,
      "ingestionLogShowDetail": false
    },
    "api": {
      "listenUri": "http://127.0.0.1:15341"
    },
    "useApiKeyForwarding": true
  },
  "encryption": {
    "encryptor": null,
    "encryptorArgs": null,
    "decryptor": null,
    "decryptorArgs": null
  },
  "profiles": {}
}

Looks like this may happen once one of the clef buffers hits the targetChunkSizeBytes limit? Restarting the service will run through one chunk and then fail to delete it. Maybe a leaked file handle or something?

[liammclennan]

Hi @chuseman

Have you tried excluding 'C:\ProgramData\Seqcli' from your antivirus scanning?

[chuseman]

Have you tried excluding 'C:\ProgramData\Seqcli' from your antivirus scanning?

No, we don't have any exclusions setup. We run Crowdstrike and that's corporate controlled so it would take some doing to get an exclusion added.

Is the behavior here different from the old seqfwd service? That ran fine on the same servers -- we've had to roll back to it for now.

[nblumhardt]

The old Seq Forwarder (essentially) used a single file that it would lock at start-up, so there would be minimal opportunities for AV scanners etc. to jump in and cause locking problems. The new version uses a set of rolling log files, so would be more susceptible to this kind of outside interference.

There's also a chance it's a lurking bug we haven't yet spotted. You might be able to get some more info on which process is using the lock via resmon or similar.