Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AI/ML platform for Jaeger #1639

Open
pavolloffay opened this issue Jun 27, 2019 · 18 comments
Open

AI/ML platform for Jaeger #1639

pavolloffay opened this issue Jun 27, 2019 · 18 comments

Comments

@pavolloffay
Copy link
Member

@pavolloffay pavolloffay commented Jun 27, 2019

Placeholder issue for any discussion related to ML/AI integration with Jaeger. On the recent Jaeger bi-weekly meetings we have talked about doing ML/AI on tracing data (and also with combination with other telemetry data like metrics and logs).

At the moment doing ML/AI analysis with Jaeger is hard. There is no direct integration with ML/AI platforms and we do not have much knowledge on what models we could build.

My objective is here is to create:

  • community/SIG for doing ML/AI with tracing/telemetry data.
  • ML/AI integration with Jaeger to make it easy for data scientists write and evaluate models (e.g jupyter notebooks).
  • create a registry of models/post-processing pipelines which derive useful information out of tracing data.

For the completion, I will list existing ML/post-processing integrations:

cc) @annanay25

@pavolloffay pavolloffay pinned this issue Jun 27, 2019
@pavolloffay

This comment has been minimized.

Copy link
Member Author

@pavolloffay pavolloffay commented Jun 27, 2019

As a first step, we should gather people interested in this to drive the right decisions. cc @jaegertracing/jaeger-maintainers

Secondly, we should start working on the integration to make it easy to start writing models. It seems there are two main WEB based platforms: jupyter and zeppelin. Both have pros and cons:

Jupyter

Zeppelin

@pavolloffay

This comment has been minimized.

Copy link
Member Author

@pavolloffay pavolloffay commented Jun 27, 2019

If you are interested comment on this issue or send me PM on gitter and I will add you to https://github.com/orgs/jaegertracing/teams/data-analytics

@pavolloffay

This comment has been minimized.

Copy link
Member Author

@pavolloffay pavolloffay commented Jul 16, 2019

Our next steps could be to try Jupyter with java/scala kernel and make connections to our DB/kafka.

@yurishkuro yurishkuro unpinned this issue Aug 1, 2019
@yurishkuro yurishkuro pinned this issue Aug 1, 2019
@annanay25

This comment has been minimized.

Copy link
Member

@annanay25 annanay25 commented Sep 22, 2019

After a discussion with Pavol, we have decided to work top-down by first compiling a list of high level objectives that we want to achieve using the AI/ML analysis. We could then gauge interest in the community about the most helpful features, and prioritise accordingly.

Once our targets are clear, not only will it help us define a clear path for development but also encourage contribution from folks with more knowledge on building data analysis models.

cc @jaegertracing/data-analytics

@pavolloffay

This comment has been minimized.

Copy link
Member Author

@pavolloffay pavolloffay commented Sep 23, 2019

Anybody is welcome to propose/upvote any feature which would help us with this initiative.

The objectives from the initial comments still hold. First we would like to build a community of people who would like to contribute (models, integrations), validate models. Secondly provide AI/ML integration as part of the upstream project. This should ultimately result in new features added to Jaeger main distributions and UI interface.

To be able to start working on the models we should provide an environment to do that. Specifically I am talking about Jupyter notebook integration with Jaeger. Provide a notebook with spark/flink connected to Jaeger data storages.

@pavolloffay

This comment has been minimized.

Copy link
Member Author

@pavolloffay pavolloffay commented Sep 23, 2019

@yurishkuro also proposed to create graph query language (similar to canopy's capabilities) which would allow defining graph related queries.

Screenshot of Jaeger Project Bi-Weekly Call - Google Docs

https://research.fb.com/publications/canopy-end-to-end-performance-tracing-at-scale

@pavolloffay

This comment has been minimized.

Copy link
Member Author

@pavolloffay pavolloffay commented Sep 23, 2019

I would like to hear @jaegertracing/data-analytics opinion on which language they would like to use for data analytics with Jaeger. Would it be Java or python?

@yurishkuro

This comment has been minimized.

Copy link
Member

@yurishkuro yurishkuro commented Sep 23, 2019

I would start with Python, it is the de-facto DS/ML language. We can later extend it to Java if necessary.

@Talina06

This comment has been minimized.

Copy link
Member

@Talina06 Talina06 commented Sep 23, 2019

I would like to hear @jaegertracing/data-analytics opinion on which language they would like to use for data analytics with Jaeger. Would it be Java or python?

I prefer Python because it's easier to get off the ground, also as @yurishkuro mentioned, it provides good libraries for DS/ML purpose.

To be able to start working on the models we should provide an environment to do that. Specifically I am talking about Jupyter notebook integration with Jaeger. Provide a notebook with spark/flink connected to Jaeger data storages.

I would be interested in starting with the Jupyter integration with Jaeger. Maybe once we have this in place, gathering requirements for building models could be easier. What do you suggest @pavolloffay?

@pavolloffay

This comment has been minimized.

Copy link
Member Author

@pavolloffay pavolloffay commented Sep 23, 2019

Ack for python, so let's start this :).

@Talina06 this is great. I will try to summarize requirements I can think of:

  1. run the jupyter/jupyterlab as docker container

  2. have a notebook file with basic connector to the storage - e.g. Elasticsearch or do streaming with Kafka.

  3. the connector might depend on the framework we choose - we could start with spark or flink (both support python). I would like to also hear what people prefer here.

@Talina06

This comment has been minimized.

Copy link
Member

@Talina06 Talina06 commented Sep 23, 2019

Ack for python, so let's start this :).

@Talina06 this is great. I will try to summarize requirements I can think of:

  1. run the jupyter/jupyterlab as docker cotainer
  2. have a notebook file with basic connector to the storage - e.g. Elasticsearch or do streaming with Kafka.
  3. the connector might depend on the framework we choose - we could start with spark or flink (both support python). I would like to also hear what people prefer here.

Sounds good. Let me get started with Spark in the meantime and share an update here.

@VishvendraRana

This comment has been minimized.

Copy link

@VishvendraRana VishvendraRana commented Sep 23, 2019

  1. the connector might depend on the framework we choose - we could start with spark or flink (both support python). I would like to also hear what people prefer here.

@pavolloffay IMO, we should start working on connector for spark. Spark has a larger community base and is more widely used tool for data analysis..

@yurishkuro

This comment has been minimized.

Copy link
Member

@yurishkuro yurishkuro commented Sep 23, 2019

I would say we should start with the library/DSL for writing query and analysis, not with Spark integration. A library is useful on its own as there could be many different sources of traces it would work with in Jupyter, like loading from a file or from query-service.

@pavolloffay

This comment has been minimized.

Copy link
Member Author

@pavolloffay pavolloffay commented Sep 24, 2019

We can start simultaneously with both. Both could be useful for different use-cases.

I have created a separate issue for DSL #1811.

Issue for jupyter notebook #1813 - cc) @Talina06

@yurishkuro

This comment has been minimized.

Copy link
Member

@yurishkuro yurishkuro commented Sep 25, 2019

I replied in #1811 (comment)

We don't need a full blown DSL, just a data model of a trace as a graph. Once we have that, people can start writing jupyter scripts.

@pavolloffay

This comment has been minimized.

Copy link
Member Author

@pavolloffay pavolloffay commented Oct 10, 2019

We should move our protos to ild repository and allow building to other languages:#1213. That will be required to consume data from Kafka.

@pavolloffay

This comment has been minimized.

Copy link
Member Author

@pavolloffay pavolloffay commented Oct 10, 2019

It would also help if the compiled model classes were published as artifacts. Now if somebody wants to consume data from kafka it requires a lot of additional work to be done.

@pavolloffay

This comment has been minimized.

Copy link
Member Author

@pavolloffay pavolloffay commented Dec 20, 2019

I have moved my POC with trace DSL using gremlin and packaged in jupyter notebook to https://github.com/jaegertracing/jaeger-analytics-java.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.