Add span_id attribute to Events (instrumentation)

[nerdai]

Description

• This PR aims to associate an Event with the Span enclosure where it takes place
• Specifically a span_id field is added to BaseEvent
• Coroutine safety is handled by assigning a current_span_id to dispatcher which we use asyncio.Lock to update when working with async funcs
• When firing an event with dispatcher, we first get dispatcher.current_span_id and store it into a variable that can be re-used within the scope of the func
• Also added a SpanDropEvent that fires if a Span is being dropped due to an encountered Exception.
  • First image: BaseQueryEngine.aquery span IS NOT dropped so QueryEndEvent is observed
  • Second image: BaseQueryengine.aquery span IS dropped so SpanDropEvent is observed and QueryEndEvent is not

Fixes # (issue)

New Package?

Did I fill in the tool.llamahub section in the pyproject.toml and provide a detailed README.md for my new integration or package?

• Yes
• No

Version Bump?

Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

• Yes
• No

Type of Change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

• Added new unit/integration tests
• Ran notebooks (that tests end-to-end)
• I stared at the code and made sure it makes sense

[logan-markewich]

rather than getting the span-id and passing it in, could event() attach this automatically?

[logan-markewich]

(would make development easier and less error-prone)

[logan-markewich]

This is actually why the callback manager used a context manager
with callback_manager.event() as event:

So if the dispatcher needs a similar pattern, we could use that again, or if possible, it just gets handled automatically

[nerdai]

That's a good point. We could maybe try to adopt a context manager approach:

with dispatcher.span_session() as dispatch_event:
     dispatch_event()
     # perform other logic
     dispatch_event()

So, here I think we add span_id arg to event() i.e., dispatcher.event(span_id, event). Thus, span_id will added to the event within .event(). And then make it so that the context manager yields a partial of dispatcher.event(span_id, event) i.e., dispatcher_event = partial(dispatcher.event, span_id=dispatcher.current_span_id)

# dispatcher.py
def event(event: BaseEvent, span_id: str = ""):
     # modify span_id field of event
     event.span_id = span_id
     ...

[logan-markewich]

Since we already decorated the function, I wonder if we can do something like this
https://stackoverflow.com/questions/12435765/can-a-decorated-function-access-variables-of-the-decorator

It would completely hide this from the implementation (and maintains a cleaner look to the code for readability)

Thoughts?

[logan-markewich]

This way, when .event() is called, it can automatically pull the span ID from globals? Maybe? (if the injected global is a function/lambda?)

[logan-markewich]

If that doesn't work, then context manager it is (even though its slightly less neat)

[nerdai]

For visibility (lol we're discussing via slack on this too -- my b):

I think this will work! Attach the attr to the wrapper which should still be coroutine safe I think. Will give it a shot.

[nerdai]

alright, wrapt complicated things for me a bit. Resorted to using context manager approach as described previously in this thread.