Implement graceful shutdown with pipeline draining

[tigrannajaryan]

Currently when shutting down any data that is in the pipeline is lost. We need to drain the pipeline during shutdown. First we need to shutdown receivers, then processors (and implement draining in processors that stores data), then exporters.

[joe-elliott]

I may be misunderstanding the need, but except for the processors this appears to be done:

opentelemetry-collector/service/service.go

Lines 304 to 317 in b468dee

	func (app *Application) shutdownPipelines() {
	// Shutdown order is the reverse of building: first receivers, then flushing pipelines
	// giving senders a chance to send all their data. This may take time, the allowed
	// time should be part of configuration.
	app.logger.Info("Stopping receivers...")
	app.builtReceivers.StopAll()
	// TODO: shutdown processors by calling Shutdown() for each processor in the
	// order they are arranged in the pipeline.
	app.logger.Info("Shutting down exporters...")
	app.exporters.ShutdownAll()
	}

[tigrannajaryan]

except for the processors

Yes, this is the missing part. Processors support partially coming in #488

We still need to implement draining in processors (particularly in batch and queue).

[jan25]

Hi! Can i work on this?

After quick look at queued_processor, this is a rough idea to fix - On shutdown have a deadline of 5s or until queue is emptied(whichever happens first). Also, We can have stats on how many spans were flushed and dropped during shutdown process. WDYT?

I see there is Stop() in queued_processor, any hints where is that used?

Similar idea can be applied to batch_processor too.

[tigrannajaryan]

@jan25 do you still want to work on this?

We will need to drain the pipeline by first stopping receivers, then stopping processors in the right order, then stop exporters. When stopping we will need to wait for up to certain max time, you are right. It needs to be done for all components, not just certain processors.

[jan25]

@tigrannajaryan Sure, I'll start the work and open a PR soon

[tigrannajaryan]

@jan25 I will remove this from 1.0 milestone, since it is not a must-have, but feel free to work on it. We will accept it when it is ready.

[kamalmarhubi]

@tigrannajaryan Is there a suggested workaround to avoid dropping messages during collector upgrades?

[tigrannajaryan]

There is no known workaround.

[kamalmarhubi]

@tigrannajaryan in that case, I feel like this should be added back to the GA milestone? Dropping things on the floor during upgrades is not great...

[tigrannajaryan]

@kamalmarhubi Help is wanted on this one. If there are contributors who can submit a PR before GA, we will be happy to review it.

[bogdandrutu]

I think this is done correct?