🚀 Feature: re-write LlamaIndex instrumentation to use LlamaIndex CallbackManager

[nirga]

Which component is this feature for?

LlamaIndex Instrumentation

🔖 Feature description

Right now, we monkey-patch classes and methods in LlamaIndex which requires endless work and constant maintenance. LlamaIndex has a system for callbacks that can potentially be used to create/end spans without being too coupled with with the framework's inner structure.

🎤 Why is this feature needed ?

Support LlamaIndex entirely and be future-proof to internal API changes

✌️ How do you aim to achieve this?

Look into LlamaIndex callback_manager and how other frameworks are using it.

🔄️ Additional Information

No response

👀 Have you spent some time to check if this feature request has been raised before?

• I checked and didn't find similar issue

Are you willing to submit PR?

None

[kartikay-bagla]

@quest-bot embark

[quest-bot]

⚠️ There's no active Quest for this Issue.

Check the docs for more info.

[kartikay-bagla]

@nirga I was looking at the LlamaDebugHandler (in the python code) and it looks like in terms of features, I just need to recreate this and extend it to add the existing traces. Does that sound correct?

[nirga]

Yep @kartikay-bagla. It's a bit trickier cause you need to create and close spans accordingly, and make sure to nest them correctly.

[kartikay-bagla]

Yeah, I'll get started on this because I think it'll take some time.

[kartikay-bagla]

Small update, with this being a larger code change than expected, and some unscheduled life things, I might have to put this on hold until the 18th. Will keep you updated though.

[kartikay-bagla]

So I was able to make some progress over the weekend, but it is slow as I couldn't find any existing implementation of callback_manager that I could use as a starting point.

[nirga]

Thanks so much @kartikay-bagla! I've been in touch with the folks at LlamaIndex so ping me here or on slack if you have any questions I can pass along.

[JCaraballo113]

Hi, I will try to solve this issue. I do have one question. This feature is desired for the JS SDK or the Python SDK or both?

[nirga]

For both @JCaraballo113 :)
@kartikay-bagla are you still on this?