Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Guidance on how to log to a message queue for slow data stores #11801

Closed
Fniz opened this issue Apr 3, 2019 — with docs.microsoft.com · 11 comments · Fixed by #15865
Closed

Guidance on how to log to a message queue for slow data stores #11801

Fniz opened this issue Apr 3, 2019 — with docs.microsoft.com · 11 comments · Fixed by #15865

Comments

Copy link

@Fniz Fniz commented Apr 3, 2019 — with docs.microsoft.com

Can you give me more precision regarding this sentence : "If your logging data store is slow, don't write to it directly. Consider writing the log messages to a fast store initially, then move them to the slow store later. For example, log to a message queue that's read and persisted to slow storage by another process."

I want to store log in SQL server, so I suppose it's a slow data store. So I would like what you mean by "log to a message queue". Do you reffer to RabbitMQ for example ?

Yours sincerely,


Document Details

Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

@Rick-Anderson

This comment has been minimized.

Copy link
Contributor

@Rick-Anderson Rick-Anderson commented Apr 4, 2019

@tdykstra do you want to take this issue and update the article to say
Log to a message queue (such as ...)

@Rick-Anderson Rick-Anderson added the P2 label Apr 4, 2019
@Rick-Anderson Rick-Anderson added this to the Backlog milestone Apr 4, 2019
@tdykstra

This comment has been minimized.

Copy link
Contributor

@tdykstra tdykstra commented Apr 4, 2019

@Rick-Anderson Yes, good idea.

@Fniz

This comment has been minimized.

Copy link
Author

@Fniz Fniz commented Apr 4, 2019

Thank you very much for your answer. Is there non-cloud credible alternative to Azure service bus ?

@Fniz Fniz closed this Apr 4, 2019
@anurse

This comment has been minimized.

Copy link
Member

@anurse anurse commented Apr 4, 2019

Actually this refers to using in-memory queuing. If you are logging to SQL server, you don't want to do that directly in the Log method, since it's synchronous. Instead, you should (synchronously) add the log message to an in-memory queue and have a background worker come along to pull the items out of the queue and do the asynchronous work (pushing data to the SQL server).

@tdykstra tdykstra added this to To Do in April 2019 (4/1 to 4/28) via automation Apr 4, 2019
@tdykstra tdykstra modified the milestones: Backlog, Sprint Apr 4, 2019
@Fniz

This comment has been minimized.

Copy link
Author

@Fniz Fniz commented Apr 8, 2019

Thank you @anurse for your answer. When you talk about an in-memory queue, is it like this for the ConsoleLogger : https://github.com/aspnet/Logging/blob/master/src/Microsoft.Extensions.Logging.Console/Internal/ConsoleLoggerProcessor.cs

@anurse

This comment has been minimized.

Copy link
Member

@anurse anurse commented Apr 8, 2019

@Fniz precisely right! Even the console logger ended up finding improvements by using an in-memory queue. A SQL (or other remote) backend would definitely want something similar.

@anurse

This comment has been minimized.

Copy link
Member

@anurse anurse commented Apr 8, 2019

FYI: Rather than a special thread, I'd suggest looking at IHostedService to process the queued log messages. It provides a built-in way to have long-running background tasks in ASP.NET Core. The console logger is more general-purpose (designed to work outside ASP.NET Core) so it uses a custom thread, but if you are in an ASP.NET Core application, IHostedService will cooperate better with the rest of the application.

@tdykstra tdykstra moved this from To Do to In progress in April 2019 (4/1 to 4/28) Apr 8, 2019
@tdykstra tdykstra moved this from In progress to Done in April 2019 (4/1 to 4/28) Apr 9, 2019
@wadebee

This comment has been minimized.

Copy link

@wadebee wadebee commented Apr 16, 2019

I know this issue is closed but it seems like a common use-case to want to log to a database. In the documentation can you point users to a 3rd party .NET Core logging provider that can queue up async logging?

@tdykstra tdykstra reopened this Apr 16, 2019
April 2019 (4/1 to 4/28) automation moved this from Done to In progress Apr 16, 2019
@tdykstra tdykstra moved this from In progress to Next up in April 2019 (4/1 to 4/28) Apr 16, 2019
@tdykstra

This comment has been minimized.

Copy link
Contributor

@tdykstra tdykstra commented Apr 18, 2019

@anurse is there a 3rd party provider we can recommend? Or sample code for a queuing provider that uses IHostedService that we could point to?

@tdykstra tdykstra moved this from Next up to In progress in April 2019 (4/1 to 4/28) Apr 18, 2019
@anurse

This comment has been minimized.

Copy link
Member

@anurse anurse commented Apr 18, 2019

That uses IHostedService? No. The console logger does this but it uses a background thread instead. IHostedService does require that you are in a "hosted" environment (either ASP.NET Core or the new Generic Host stuff) so perhaps a background thread may be a better option for a logger provider since logger providers can be used in arbitrary class libraries, apps, etc.

@tdykstra tdykstra changed the title More precision on "Log to a message queue" meaning Guidance on how to log to a message queue for slow data stores Apr 19, 2019
@tdykstra

This comment has been minimized.

Copy link
Contributor

@tdykstra tdykstra commented Apr 19, 2019

I changed the title to indicate this issue is about guidance for writing (and/or sample code for) a logging provider that uses in-memory queue for fast logging and uses a background process to pull from the queue to write to a slow data store. This is related to, and could be done along with, #11842

@tdykstra tdykstra added P1 PU labels Apr 19, 2019
@tdykstra tdykstra removed their assignment Apr 19, 2019
@tdykstra tdykstra removed this from In progress in April 2019 (4/1 to 4/28) Apr 19, 2019
@tdykstra tdykstra modified the milestones: Sprint, Backlog Apr 19, 2019
@Rick-Anderson Rick-Anderson self-assigned this Nov 26, 2019
@Rick-Anderson Rick-Anderson added this to To do in November 2019 via automation Nov 26, 2019
November 2019 automation moved this from To do to Done Nov 26, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
November 2019
  
Done
7 participants
You can’t perform that action at this time.