Refactored LogRecords to be mutable so that attributes can be modified after creation #131

evanlauer1 · 2024-06-12T19:49:38Z

Big Context

Logging Support is being added to hs-opentelemetry. Logging spec

Small (This PR) Context

LogRecords should be modifiable after creation for the purposes of LogRecordProcessors. This PR refactors LogRecord to contain a mutable IORef to an ImmutableLogRecord. It also adds convenience functions for adding attributes to LogRecords and a test suite that tests LoggerProviders and LogRecords.

LogRecords can only be read through the functions of ReadableLogRecord and modified by the functions of ReadWriteLogRecord. This is in accordance with the spec.

The type of the body field was changed from a type variable to an AnyValue so that processors and exporters can process logs from various sources that might have different body types.

Testing

stack build runs
New test suite under stack test hs-opentelemetry passes

evanlauer1 · 2024-06-12T19:49:57Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @evanlauer1 and the rest of your teammates on Graphite

evanlauer1

Finished self-review

evanlauer1 · 2024-06-12T19:57:10Z

api/src/OpenTelemetry/Internal/Logging/Types.hs

+data LogRecord a = LogRecord (IORef (ImmutableLogRecord a))
+
+
+data ImmutableLogRecord body = ImmutableLogRecord


The LogRecord containing an IORef to an ImmutableLogRecord mirrors the pattern of a Span containing an ImmutableSpan.

Ah, sounds like there's precedence for this.

Upon reading the description, I was wondering about implementing a more functional approach wherein each LogRecord could optionally reference a previous log record and they would be merged/folded upon consumption. But I would go with what Ian did on the tracing side.

I am curious about the reason Ian used an IORef instead of something like the State monad. I'll check with him about that when he gets back.

As a thought exercise, consider adding logging to an IO function. You have a Logger, you create a LogRecord body value.

You do some IO and want to add some attributes to the LogRecord body, then emit the log. Usage of the state monad doesn't work for that unless you want to force people to rewrite their monad stack, which we don't.

With regards to why we use an IORef, it's because we're more or less told that we have to do it that way: https://opentelemetry.io/docs/specs/otel/logs/sdk/#additional-logrecord-interfaces

In that regard, we probably want to update our naming here to reflect the naming outlined in that section.

That makes sense. In regards to naming, Spans also have "readable" and "read/write" versions defined in the spec, but hs-opentelemetryonly implements the read/write spans. The spec says that readable log records can be implemented as read/write log records instead. Essentially, since LogRecord is both a ReadableLogRecord and ReadWriteLogRecord, I thought it would be simplest to call it LogRecord. It would be possible to do both, but ReadableLogRecord would have to be somethings like

module Readable ( ReadableLogRecord, -- not exporting constructor readReadableLogRecord ) where -- It needs to be an IORef because it has to contain all changes made to the LogRecord data ReadableLogRecord body = ReadableLogRecord (IORef (ImmutableLogRecord body)) readReadableLogRecord :: ReadableLogRecord body -> IO (ImmutableLogRecord body)

I ended up writing a typeclass for ReadableLogRecords and ReadWriteLogRecords.

evanlauer1 · 2024-06-12T19:58:03Z

api/src/OpenTelemetry/Internal/Logging/Types.hs

@@ -113,6 +118,7 @@ data LogRecord body = LogRecord
  -- ^ Additional information about the specific event occurrence. Unlike the Resource field, which is fixed for a particular source, Attributes can vary for each occurrence of the event coming from the same source.
  -- Can contain information about the request context (other than Trace Context Fields). The log attribute model MUST support any type, a superset of standard Attribute, to preserve the semantics of structured attributes
  -- emitted by the applications. This field is optional.
+  , logRecordLogger :: Logger


LogRecord contains a Logger so that it can access LoggerProvider configuration. This is currently used for attribute limits.

Something about this feels a little funny, but I don't have enough context to definitively state why at the moment. Will probably have to see how this progresses.

It does seem a bit odd. LoggerProvider stores the configuration for LogRecords, but some fields of LoggerProvider are already duplicated in LogRecord like resources so there might be a more consistent way to manage this configuration. ImmutableSpans have a reference to their parent Tracers, but they use that to access processors as well as attribute limits so the use case does not perfectly match.

evanlauer1 · 2024-06-12T19:59:12Z

api/src/OpenTelemetry/LogAttributes.hs

+  -- * Attribute limits
+  AttributeLimits (..),
+  defaultAttributeLimits,
+


LogAttributes.hs now exports AttributeLimits so that you don't need to import both OpenTelemetry.LogAttributes and OpenTelemetry.Attributes to use all of this module's functionality.

evanlauer1 · 2024-06-12T20:00:57Z

api/src/OpenTelemetry/Logging/Core.hs

 import OpenTelemetry.Common
 import OpenTelemetry.Context
 import OpenTelemetry.Context.ThreadLocal
 import OpenTelemetry.Internal.Common.Types
 import OpenTelemetry.Internal.Logging.Types
 import OpenTelemetry.Internal.Trace.Types
-import OpenTelemetry.LogAttributes
+import OpenTelemetry.LogAttributes (ToValue)
+import qualified OpenTelemetry.LogAttributes as A


Changed to a qualified import so that addAttributes does not cause a naming conflict. It would be possible to rename addAttributes to logRecordAddAttributes, but OpenTelemetry.Trace.Core. addAttributes is not named spanAddAttributes.

evanlauer1 · 2024-06-12T20:02:31Z

api/src/OpenTelemetry/Logging/Core.hs

-  -> m (LogRecord body)
-emitLogRecord Logger {..} LogRecordArguments {..} = do
+  -> m (ImmutableLogRecord body)
+createImmutableLogRecord logger@Logger {..} LogRecordArguments {..} = do


createImmutableLogRecord is essentially the same function that emitLogRecord was before. This is a helper function to separate the IORef logic from the LogRecord creation logic.

evanlauer1 · 2024-06-12T20:04:04Z

api/src/OpenTelemetry/Logging/Core.hs

+                  k
+                  v
+            }
+      )


Mirrors OpenTelemetry.Trace.Core.addAttribute

evanlauer1 · 2024-06-12T20:04:22Z

api/src/OpenTelemetry/Logging/Core.hs

+                  logRecordAttributes
+                  attrs
+            }
+      )


Mirrors OpenTelemetry.Trace.Core.addAttributes

evanlauer1 · 2024-06-12T20:04:48Z

api/src/OpenTelemetry/Logging/Core.hs

+ instrumentation.
+-}
+logRecordGetAttributes :: (MonadIO m) => LogRecord a -> m A.LogAttributes
+logRecordGetAttributes (LogRecord lr) = liftIO $ logRecordAttributes <$> readIORef lr


Mirrors OpenTelemetry.Trace.Core.spanGetAttributes

evanlauer1 · 2024-06-12T20:04:57Z

api/src/OpenTelemetry/Resource.hs

@@ -176,6 +176,7 @@ data MaterializedResources = MaterializedResources
  { materializedResourcesSchema :: Maybe String
  , materializedResourcesAttributes :: Attributes
  }
+  deriving (Show, Eq)


Used for testing

evanlauer1 · 2024-06-12T20:06:27Z

api/test/OpenTelemetry/Logging/CoreSpec.hs

+          [ ("anotherThing", "another thing")
+          , ("something", "a thing")
+          , ("twoThing", "the second another thing")
+          ]


The it clauses in this document are rather long, so I wonder if the tests could be separated into smaller parts. As it is, if a test fails you would have to muck around in these lines of code to find which line failed, but at least you will know something is wrong.

They are long, but it does seem like one behavior is being described and expected/asserted, which is good.

I could see there being a shorter way to construct LogRecordArguments... essentially a emptyLogRecordArguments that you use the record modification syntax on like:

emptyLogRecordArguments { attributes = H.fromList [ ... ] }

Done. The way I did it, emptyLogRecordArguments takes an argument of the body type, so the syntax looks more like (emptyLogRecordArguments body) {attributes = ..} which is not as clean but I don't know of a better way to do it.

andrewsmith · 2024-06-13T04:40:10Z

api/src/OpenTelemetry/Internal/Logging/Types.hs

+data LogRecord a = LogRecord (IORef (ImmutableLogRecord a))
+
+
+data ImmutableLogRecord body = ImmutableLogRecord


Ah, sounds like there's precedence for this.

Upon reading the description, I was wondering about implementing a more functional approach wherein each LogRecord could optionally reference a previous log record and they would be merged/folded upon consumption. But I would go with what Ian did on the tracing side.

andrewsmith · 2024-06-13T04:47:39Z

api/src/OpenTelemetry/Internal/Logging/Types.hs

@@ -113,6 +118,7 @@ data LogRecord body = LogRecord
  -- ^ Additional information about the specific event occurrence. Unlike the Resource field, which is fixed for a particular source, Attributes can vary for each occurrence of the event coming from the same source.
  -- Can contain information about the request context (other than Trace Context Fields). The log attribute model MUST support any type, a superset of standard Attribute, to preserve the semantics of structured attributes
  -- emitted by the applications. This field is optional.
+  , logRecordLogger :: Logger


Something about this feels a little funny, but I don't have enough context to definitively state why at the moment. Will probably have to see how this progresses.

api/test/OpenTelemetry/Logging/CoreSpec.hs

andrewsmith · 2024-06-13T05:03:07Z

api/test/OpenTelemetry/Logging/CoreSpec.hs

+          [ ("anotherThing", "another thing")
+          , ("something", "a thing")
+          , ("twoThing", "the second another thing")
+          ]


They are long, but it does seem like one behavior is being described and expected/asserted, which is good.

I could see there being a shorter way to construct LogRecordArguments... essentially a emptyLogRecordArguments that you use the record modification syntax on like:

emptyLogRecordArguments { attributes = H.fromList [ ... ] }

evanlauer1

Finished self-review for ReadableLogRecord and ReadWriteLogRecord.

evanlauer1 · 2024-07-03T18:38:23Z

api/src/OpenTelemetry/Internal/Logging/Types.hs

+  -- ^ Describes the source of the log, aka resource. Multiple occurrences of events coming from the same event source can happen across time and they all have the same value of Resource.
+  -- Can contain for example information about the application that emits the record or about the infrastructure where the application runs. Data formats that represent this data model
+  -- may be designed in a manner that allows the Resource field to be recorded only once per batch of log records that come from the same source. SHOULD follow OpenTelemetry semantic conventions for Resources.
+  -- This field is optional.


Moved here because Resource was removed from ImmutableLogRecord due to its absence from ReadWriteLogRecord in the spec.

evanlauer1 · 2024-07-03T18:39:38Z

api/src/OpenTelemetry/Internal/Logging/Types.hs

@@ -43,8 +53,76 @@ data Logger = Logger

 {- | This is a data type that can represent logs from various sources: application log files, machine generated events, system logs, etc. [Specification outlined here.](https://opentelemetry.io/docs/specs/otel/logs/data-model/)
 Existing log formats can be unambiguously mapped to this data type. Reverse mapping from this data type is also possible to the extent that the target log format has equivalent capabilities.
+Uses an IORef under the hood to allow mutability.
+-}
+data LogRecord a = LogRecord Logger (IORef (ImmutableLogRecord a))


Stores a Logger for access to InstrumentationScope in the Logger, AttributeLimits in the LoggerProvider, and Resource in the LoggerProvider.

evanlauer1 · 2024-07-03T18:40:44Z

api/src/OpenTelemetry/Internal/Logging/Types.hs

 -}
-data LogRecord body = LogRecord
+class ReadableLogRecord r where


This can be used to constrain the available operations to read-only.

evanlauer1 · 2024-07-03T18:42:03Z

api/src/OpenTelemetry/Internal/Logging/Types.hs


 module OpenTelemetry.Internal.Logging.Types (
  LoggerProvider (..),
  Logger (..),
-  LogRecord (..),
+  LogRecord,


The constructor of LogRecord is no longer exported. Any interaction with LogRecord must be done through the functions of ReadableLogRecord and ReadWriteLogRecord.

evanlauer1 · 2024-07-03T18:44:58Z

api/src/OpenTelemetry/Internal/Logging/Types.hs

+
+
+  -- | Modifies the @LogRecord@ using its internal @IORef@. This is lazy and is not an atomic operation. The implementation mirrors @modifyIORef@.
+  modifyLogRecord :: r a -> (ImmutableLogRecord a -> ImmutableLogRecord a) -> IO ()


There is currently no strict version of modifyLogRecord. It might be better to make modifyLogRecord by default strict and export no lazy version, but I could also write both modifyLogRecord and modifyLogRecord'.

evanlauer1 · 2024-07-03T18:46:40Z

api/src/OpenTelemetry/Internal/Logging/Types.hs

-  -- Can contain for example information about the application that emits the record or about the infrastructure where the application runs. Data formats that represent this data model
-  -- may be designed in a manner that allows the Resource field to be recorded only once per batch of log records that come from the same source. SHOULD follow OpenTelemetry semantic conventions for Resources.
-  -- This field is optional.
-  , logRecordInstrumentationScope :: InstrumentationLibrary


These should not be able to be changed by modifyLogRecord so they are removed from ImmutableLogRecord. They are now contained in the Logger of LogRecord.

evanlauer1

Finished self-review

evanlauer1 · 2024-07-12T03:23:01Z

api/src/OpenTelemetry/Internal/Common/Types.hs

+  | ArrayValue [AnyValue]
+  | HashMapValue (H.HashMap Text AnyValue)
+  deriving stock (Read, Show, Eq, Ord, Data, Generic)
+  deriving anyclass (Hashable)


Moved here from LogAttributes.hs because it is now also used in LogRecord.

evanlauer1 · 2024-07-12T03:25:40Z

api/src/OpenTelemetry/Internal/Logging/Types.hs

@@ -92,7 +170,7 @@ data LogRecord body = LogRecord
  --
  -- In the contexts where severity participates in less-than / greater-than comparisons SeverityNumber field should be used.
  -- SeverityNumber can be compared to another SeverityNumber or to numbers in the 1..24 range (or to the corresponding short names).
-  , logRecordBody :: body
+  , logRecordBody :: AnyValue


Replaced the body type variable with AnyValue so that if logs come in from different sources with different body types, processors and exporters can still process them. body is defined here in the spec.

evanlauer1

Finished self-review

evanlauer1 · 2024-07-15T19:47:18Z

api/src/OpenTelemetry/Internal/Logging/Types.hs

+mkReadWriteLogRecord l = fmap (ReadWriteLogRecord l) . newIORef
+
+
+newtype ReadableLogRecord = ReadableLogRecord {readableLogRecord :: ReadWriteLogRecord}


Changed to be a concrete type so that it can be used in data LogRecordExporter.

evanlauer1 · 2024-07-15T19:47:42Z

api/src/OpenTelemetry/Internal/Logging/Types.hs

 -}
-data LogRecord body = LogRecord
+data ReadWriteLogRecord = ReadWriteLogRecord Logger (IORef ImmutableLogRecord)


Changed to a concrete type to mirror ReadableLogRecord

evanlauer1

Finished self-review

evanlauer1 · 2024-07-22T19:42:36Z

api/src/OpenTelemetry/Internal/Common/Types.hs

+  | ByteStringValue ByteString
+  | ArrayValue [AnyValue]
+  | HashMapValue (H.HashMap Text AnyValue)
+  | NullValue


Added NullValue in accordance with the spec.

evanlauer1 · 2024-07-22T19:44:18Z

api/src/OpenTelemetry/Internal/Logging/Types.hs

-emptyLogRecordArguments :: body -> LogRecordArguments body
-emptyLogRecordArguments body =
+emptyLogRecordArguments :: LogRecordArguments
+emptyLogRecordArguments =


Removed body because of the addition of NullValue and the removal of the type parameter to LogRecordArguments

…d after creation

…ecordArguments

evanlauer1 mentioned this pull request Jun 12, 2024

Add LoggerProvider to create Loggers and allow registering a global LoggerProvider #128

Merged

evanlauer1 mentioned this pull request Jun 12, 2024

Refactored attribute limits to be controlled by the LoggerProvider #129

Merged

evanlauer1 changed the title ~~Refactored LogRecords to be mutable so that attributes can be modified after creation~~ Refactored LogRecords to be mutable so that attributes can be modified after creation Jun 12, 2024

evanlauer1 changed the title ~~Refactored LogRecords to be mutable so that attributes can be modified after creation~~ Refactored LogRecords to be mutable so that attributes can be modified after creation Jun 12, 2024

evanlauer1 commented Jun 12, 2024

View reviewed changes

evanlauer1 mentioned this pull request Jun 13, 2024

Added LogProcessors and implemented shutdown and forceFlush for LoggerProviders #132

Merged

andrewsmith reviewed Jun 13, 2024

View reviewed changes

evanlauer1 force-pushed the refactor-attribute-limits-to-logger-provider branch from 27fbd05 to bdfb7c1 Compare June 14, 2024 21:52

evanlauer1 force-pushed the refactor-log-records-to-be-mutable branch from 9aeda12 to 62fd9ea Compare June 14, 2024 21:52

evanlauer1 force-pushed the refactor-attribute-limits-to-logger-provider branch from bdfb7c1 to 8e35e04 Compare June 21, 2024 17:46

evanlauer1 force-pushed the refactor-log-records-to-be-mutable branch from 4a91f00 to c024ce3 Compare June 21, 2024 17:46

evanlauer1 mentioned this pull request Jul 2, 2024

Added Logging when creating spans or emitting logs drops attributes. Updated Severity Number #133

Merged

evanlauer1 force-pushed the refactor-log-records-to-be-mutable branch from c024ce3 to be702b6 Compare July 2, 2024 20:12

evanlauer1 changed the base branch from refactor-attribute-limits-to-logger-provider to add-emit-logs-on-dropped-attributes July 2, 2024 20:12

evanlauer1 force-pushed the refactor-log-records-to-be-mutable branch from be702b6 to 7047eee Compare July 2, 2024 21:27

evanlauer1 requested a review from iand675 July 2, 2024 22:54

evanlauer1 force-pushed the refactor-log-records-to-be-mutable branch from 7047eee to 3c7470b Compare July 3, 2024 00:35

evanlauer1 commented Jul 3, 2024

View reviewed changes

evanlauer1 requested a review from andrewsmith July 3, 2024 18:49

evanlauer1 force-pushed the add-emit-logs-on-dropped-attributes branch from 17c60da to f14afa9 Compare July 3, 2024 19:04

evanlauer1 force-pushed the refactor-log-records-to-be-mutable branch from 2dd6a5b to 36ca23f Compare July 3, 2024 19:04

evanlauer1 mentioned this pull request Jul 4, 2024

Renamed Processor to SpanProcessor, Exporter to SpanExporter, and Logging to Logs #134

Merged

evanlauer1 force-pushed the refactor-log-records-to-be-mutable branch from 36ca23f to ec5674e Compare July 4, 2024 01:20

This was referenced Jul 8, 2024

Added LogRecordExporters and updated documentation. #135

Merged

Added Simple LogRecordProcessor and refactored shutdown and forceFlush #136

Draft

evanlauer1 commented Jul 12, 2024

View reviewed changes

iand675 approved these changes Jul 15, 2024

View reviewed changes

evanlauer1 commented Jul 15, 2024

View reviewed changes

evanlauer1 force-pushed the refactor-log-records-to-be-mutable branch from 8503d1f to 76b7962 Compare July 15, 2024 21:01

evanlauer1 force-pushed the add-emit-logs-on-dropped-attributes branch from 7d6708b to 273c3c6 Compare July 15, 2024 22:55

evanlauer1 force-pushed the refactor-log-records-to-be-mutable branch from 76b7962 to 2b3cf49 Compare July 15, 2024 22:55

evanlauer1 commented Jul 22, 2024

View reviewed changes

evanlauer1 added 8 commits July 23, 2024 17:13

Refactored LogRecords to be mutable so that attributes can be modifie…

bfa7827

…d after creation

Updated test suite

662979b

Added ReadableLogRecord and ReadWriteLogRecord interfaces

f629950

Added typeclasses for ReadableLogRecords and ReadWriteLogRecords

e60ddbf

Replaced type variable in LogRecord with AnyValue

5cc35e5

Changed Readable and ReadWriteLogRecord to be concrete types

2c80ef7

Changed mkReadableLogRecord to take in a ReadWriteLogRecord

9e0cd5e

Added NullValue to AnyValue and removed body parameter from emptyLogR…

5af2e9e

…ecordArguments

evanlauer1 force-pushed the refactor-log-records-to-be-mutable branch from b6a4c70 to 5af2e9e Compare July 24, 2024 00:13

evanlauer1 changed the base branch from add-emit-logs-on-dropped-attributes to main July 24, 2024 00:13

evanlauer1 marked this pull request as ready for review July 24, 2024 00:16

evanlauer1 merged commit 0c5d2bd into main Jul 24, 2024
7 checks passed

evanlauer1 deleted the refactor-log-records-to-be-mutable branch July 24, 2024 00:23

		data LogRecord a = LogRecord (IORef (ImmutableLogRecord a))


		data ImmutableLogRecord body = ImmutableLogRecord



		-- \| Modifies the @LogRecord@ using its internal @IORef@. This is lazy and is not an atomic operation. The implementation mirrors @modifyIORef@.
		modifyLogRecord :: r a -> (ImmutableLogRecord a -> ImmutableLogRecord a) -> IO ()

		mkReadWriteLogRecord l = fmap (ReadWriteLogRecord l) . newIORef


		newtype ReadableLogRecord = ReadableLogRecord {readableLogRecord :: ReadWriteLogRecord}

Refactored LogRecords to be mutable so that attributes can be modified after creation #131

Refactored LogRecords to be mutable so that attributes can be modified after creation #131

Conversation

evanlauer1 commented Jun 12, 2024 • edited Loading

Big Context

Small (This PR) Context

Testing

evanlauer1 commented Jun 12, 2024 • edited Loading

evanlauer1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evanlauer1 Jun 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evanlauer1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evanlauer1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evanlauer1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evanlauer1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evanlauer1 commented Jun 12, 2024 •

edited

Loading

evanlauer1 commented Jun 12, 2024 •

edited

Loading

evanlauer1 Jun 21, 2024 •

edited

Loading