Embabel: LLM response processing is very slow

[mekkiamiri]

@johnsonr and @alexheifetz I implemented agents using embabel and crewAI (python). For the same use case embabel takes more much time to do the same exercice. I don't know if it is related to streaming feature which is missing or i missed some configuration.
For information I tested with java example with a custom open ai compatible LLM.
Could you please help ?
thanks

[alexheifetz]

can you share link to your github repo, so i can run the code and see what is going on?
Tnx!

[mekkiamiri]

thanks
this is the git repo: https://github.com/mekkiamiri/software-company-agents

[johnsonr]

Happy to look at the repo. There should be no difference in latency.

Would also like to see any log output. Thanks.

[mekkiamiri]

thanks
this is the git repo: https://github.com/mekkiamiri/software-company-agents

[johnsonr]

Can you please verify what LLM is being called. It looks like it it will be the auto LLM, not your custom LLM. You should see this in the logs.

Are you sure Crew and Embabel applications are calling the same model?

[mekkiamiri]

@johnsonr the model is the custom gemini 2.5 pro (. I am sure that it is the same model that i used for crew ai.
the first step to generate user stories md file takes about 5mn with embabel however with crewai it takes few seconds.
this is part of log to show the model:

`Powered by Spring Boot 3.5.5
09:47:11.751 [main] INFO  SoftwareCompanyApplication - Starting SoftwareCompanyApplication using Java 21.0.6 with PID 29593 (/Users/Documents/software-company-agents/target/classes started by d53283 in /UsersDocuments/software-company-agents)
09:47:11.752 [main] INFO  SoftwareCompanyApplication - No active profile set, falling back to 1 default profile: "default"
09:47:12.567 [main] INFO  OllamaModels - Ollama models will not be queried as the 'ollama' profile is not active
09:47:12.570 [main] INFO  DockerLocalModels - Docker local models will not be queried as the 'docker' profile is not active
09:47:12.582 [main] INFO  BedrockModels - Bedrock models will not be queried as the 'bedrock' profile is not active
09:47:12.626 [main] INFO  CustomOpenAiModels$$SpringCGLIB$$0 - Open AI compatible models are available at https://api-dev.llm.fr/proxyllm-s3/. API key is set
09:47:12.633 [main] INFO  OpenAiModels - Using custom OpenAI base URL: https://api-dev.llm.fr/proxyllm-s3/
09:47:12.758 [main] INFO  ConfigurableModelProvider - Default LLM: Custom-Gemini-2.5-Pro
Available LLMs:
	name: Custom-Gemini-2.5-Pro, provider: CustomOpenAiCompatible
Default embedding service: text-embedding-3-small`

[alexheifetz]

Hi Mekki, very nice and more importantly useful project, to automate software development process.

I will clone the repo and give it a spin. Please share the following:

1. Sample prompt I should use to test with (so we're doing similar test)
2. There is input folder and ingestion. Do I need to put some local data for ingestion?
3. Please share link to crew.ai sample you're using as a benchmark ( I would like to take a look at the crew setup )

Thanks!

[mekkiamiri]

hi @alexheifetz
1- You don't need a prompt to test my example. You run spring boot and in the shell tape the command: x "generate software"
2- as data to ingest, you can put a prd or any requirement document. You can this attached sample
prd-shortenurl.md
3- for crewai, i have clone this git repo https://github.com/crewAIInc/crewAI-examples/tree/main/crews/game-builder-crew and changed the first agent with the same prompt in my java example.
I will be happy if we can put my example as java example in your repository. I am open to enhance and add tests.

[alexheifetz]

Thanks. I will take a closer look at what is taking time....
Please go ahead and raise a PR with software company agent. It will be a great addition to embabel-examples repo.

[mekkiamiri]

@alexheifetz thanks. I am happy to hear that from you. i finish some enhancement and i will create the pr next week

[igordayen]

@mekkiamiri - could you please update? Do you still experience performance-related issues? Thanks, Igor