Features and Roles you wish from Devika

[franklinselva]

Hello Community!

I am starting this discussion thread to let the community share their ideas and features Devika would like to play in their software cycle.
Ideas/suggestions of any size are welcome; once, we have a good list, we will group them, prioritize them and shift them to the development phase.

Feel free to share your ideas! I'm looking forward to your contributions✨✨.

[Ahmet0691]

Essentially bugs that need to be solved:
1-) I see the first link on the screen and then an error appears. (Playwright browser)
2-) Gemini gives an error warning with code 500 or was blocked while, try and except, it works when you try again without seeing this error
3-) Gemini sometimes sends an empty plan, but the application accepts this empty plan, it may send a request again by saying wrong answer.
4-) The application does not go according to the plan, writes the code, finishes the application, does not go to the testing and reporting stages respectively.
5-) When a new feature is requested, websearching it will give better quality results, but it does not do this.
6-) While websearching, it may be going to random sites, but it has 3 rights, it can search until it reaches quality data, extracting unnecessary information will give the closest result to perfection.
7-) In the interface part, if it shows both the code editor part, the terminal part and the stage in the plan, it will be better quality than the other application.
8-) Runner part, in this part, you can see that your own application is working, but when there are errors and deficiencies, you can ask the user what they want to improve by asking the user, so a better result can emerge. Again, it would be great to be able to search the web when a new feature is desired.
9-) When a website design or an application is requested, it uses the same vevn as devika, this can cause devika to run or libraries to conflict.

[franklinselva]

That's a good set of features. Noted!

[Vaneeza-7]

[franklinselva]

That's a good one. I am aware that there are specific packages across programming languages that do this job to an extent. Can you elaborate on the level of detail you would like to see in the UML diagram? Any open-source resources that you can share if possible would be helpful.

[PierrunoYT]

Option to edit existing codebases and upload files

[franklinselva]

@PierrunoYT Noted. This should be a necessary one I believe.

[erkinalp]

Ability to act as a project triager & quick fix engineer on an existing software repository

[Ahmet0691]

1-) When an incorrect apikey is entered, a popup appears on the website and warns that the correct apikey must be entered.
2-) Devika has to fulfil the plan prepared by Gemini in stages
3-) When the code gives an error, it tries to solve it itself, but if it cannot solve it 3 times, it tries to find it by web search, especially from certain sites, and then tries to fix it according to this information. (to improve SWE-BENCH)
4-) In applications that do not cause errors when running, for example, when you run the snake game, the snake game may not work properly even if no error is generated. In such cases, it is necessary to ask the user which points are missing or incorrect and reorganise the code.
5-) Devika Instead of writing and finishing the code, the code should be run properly and the code should be made to work during the process until it is tried.

[Dizzguise]

Brave API would be a nice addition. Decent free quota and relatively cheap for unlimited. Plus AI support and all.

https://brave.com/search/api/

[obliviousz]

I'd like to suggest an approach for the runner wherein we attempt to resolve issues using AI only, potentially without prior knowledge available to the AI. What I propose and intend to implement for enhanced error resolution for the rerunner is to utilize web searches to identify errors, store them in a context or similar, and then present them to the user through prompts, asking for solutions to fix the code.

If you guys want to work on this, please assign me this task then, I'd be more than happy to contribute on this and I really want to too!
Thanks

[franklinselva]

@obliviousz Noted. are you already on discord?

[obliviousz]

Not yet. How can I join?

[PierrunoYT]

Check the Link inside the Readme.

[obliviousz]

#383

Here is the PR for this

[cocobeach]

qdrant vector database for rag.py, with custom collections for experts? #355

[stevejford1]

It would be great if it could have selenium added so it can log into accounts so I can get it create things

[erkinalp]

Hardware designer and validator

[franklinselva]

Noted. I will take a look and keep you posted.

[cocobeach]

That'd be a great feature so we can design out own NLP Asics, however I would complete the already existing architecture and have more mundane tasks like RAG working well before trying to add a Foundation module (as in Asimov) :)))

I have qdrand up and running and the code is available to hook it up, but what other files need to be amended to reflect on actions and other endpoints, I would like to implement the "experts" - including hardware designer, at least the boiler plate? If you make it modular enough we can all work on different kinds of experts according to each expertise? => If we make it systematic even non coders could develop "experts"?

Fantastic job so far anyway!! love it

[erkinalp]

For further optimisation of functions beyond the capability of a large language/media model, it could take help from a sat-solver for decision problems with boolean output and a function matcher for other problems. This step may not always be necessary and would be expensive, hence might be conditioned on a certain criteria before appealing to it.

[franklinselva]

@erkinalp I didn't expect we will get to solvers and decision-making systems in LLM this soon. Looks like you have a structured plan to work on this feature. Are you on the Discord server? Can you dm me?

[cocobeach]

I run local, so the more accurate the better, it doesn't matter if I have to wait a bit.

[Parth-Jain-2002]

I'd like to suggest a versioning system for Devika. Every project should have its own git versioning in place. This will add some of these functionalities:

1. If a change doesn't go as well, the user can guide Devika to go back to previous commit.
2. If a user wants to experiment with certain experimental features, Devika can create a new branch and add new changes to it.

If you guys want then I can work on this task. Please assign me this task in that case, I'd be more than happy to contribute to it. Thanks

[Parth-Jain-2002]

Have created a pull request for the following: #377. More suggestions and review are welcomed.

[erkinalp]

Communications and networking protocol designer and validator (give it high level requirements and it should generate a highly detailed workable specification and tests out of it)

[TimeLordRaps]

An option for the model to specify files it should ignore, like a git ignore. Currently when working with node packages after installing it tries passing the whole package-lock.json to the model, this is dangerous behavior that will lead to eaten tokens and sad users.

In relation to this just better management of code that is fed into the model. SWE-agent claims to have found 100 lines at max sent into the model seems to be the best, and obviously the more relevant those lines as a set of snippets are to the reasoning necessary to solve the prompt the better. Seems like currently we are just feeding in everything, though I could be wrong about that.

[cocobeach]

Yes that's why mage form the wasp-lang framework was so promising, it only had to add stuff in the Client/Serve/main.wasp while the whole boilerplate is generated automatically, but nobody improved it in the last 6 months. You are limited to js and react but it's quite a good solution to minimise the code the AI has to write.

[franklinselva]

@cocobeach Could you share the relevant links here?

[cocobeach]

Yes this is their repo: https://github.com/wasp-lang/wasp

It works out of the box, but there is a couple of details:

1. After npm install you can run the main app "wasp new" in the terminal it gives you various examples, the most complete is SAAS, that you can run with "wasp start" after migrating the DB, (some are already selling online for 200 bucks) , but option 5 is AI, you prompt it and it creates a wasp full stack that you can run. (I wasn't able to have the AI interface to work locally, but it works from the terminal with with "wasp new:ai" (or option 5 from wasp new) and that runs the ai bot builder, mage the UI can be tested here: https://usemage.ai/
2. Now in order to develop on the AI side you need to go into waspc and compile the haskell core with cabal.
3. The devs are very busy and wasp ai hasn't changed in 6 months but currently the AI bot uses the stack but not the boilerplate examples, so it builds a basic webapp but without anything, in my opinion, if the AI started with the SAAS example: https://github.com/wasp-lang/open-saas instead of from scratch you would have this already articulate framework setup and working:

🐝 Wasp - a full-stack React, NodeJS, Prisma framework with superpowers
🚀 Astro - Astro's lightweight "Starlight" template for documentation and blog
💸 Stripe - for products and payments
📈 Plausible or Google Analytics
🤖 OpenAI - OpenAI API integrated into the app or Replicate (coming soon 👀)
📦 AWS S3 - for file uploads
📧 SendGrid, MailGun, or SMTP - for email sending
💅 TailwindCSS - for styling
🧑‍💼 TailAdmin - admin dashboard & components for TailwindCSS

So the AI could concentrate in building the actual functionalities instead of having the insurmountable task of building them again. In the case of Devika, when you start from scatch I don;t see how it could build an actual application without a full stack base and all those moving parts, it's impossible with our current context windows. Also it seems like wasp ai is just 1 linear agent and not a team with planning, devs ops, QA etc..

Maybe if we give the AI a head-start all it will have to do is write the code for client/server/shared and main.wasp, 4 pages/dirs it needs to focus just on, nothing else.

Also OpenAI is not viable especially for testing, and now starting from the 15 you have to prepay them, clever chaps, I am off it, ollama or groq or Claude could be viable alternatives. Obviously if we can test it with local AIs there would be no problem for spending, But I have burned 20 dollars (again) in 5 minutes with SWE-Agents, and it's pretty much the sane thing with every coding agent I tried, I can't do that every time I test something.

(==> Now unrelated, for Devika, I just can't have it to run properly, it writes a bit of code then calls it a day and keeps repeating the task is finished but it is not, it looks lazy. With open-devin I have the opposite problem, it keeps writing and I can't interact with it.)

Hope this helps!

[franklinselva]

That's a detailed information. Thanks a lot!

[franklinselva]

@cocobeach I agree with your feedback on Devika.

[obliviousz]

Another feature that comes to my is, I think a "tester" agent could be a good addition for creating test cases and running the application after it is prepared.

[franklinselva]

This is an essential one.

[ohbob]

I wish It just worked. Wasn't able to setup on three machines with different os. Maybe docker containers would help.

[erkinalp]

Researcher, based on ResearchAgent, to help the coding model when a problem necessiates research and a derivation of non-obvious solution

[franklinselva]

I see there are many research agents in the field. I assume you are planning to make Devika reproduce acceptable research papers for its results?

[erkinalp]

Rather, it should be able to read and translate into algorithm those research papers. Not all algorithms are obvious.

[Okrim007]

It would be very useful if Devika could run locally installed software, if necessary, in some isolated environment (Docker, etc.).

One scenario: Devika, with the assistance of the user, models in a chosen logical formalism/logical programming language, a semi-structured system of business rules, originally expressed in limited spoken English. The formalized system must be consistent, open to adding and updating new rules and deleting existing ones. It would have to act as a so-called "truth maintenance system".

In any case, the user could check the results of the interactive modeling of business rules in cooperation between Devika and the user "manually", but this would take much more time and be prone to errors, as well as fragmentation of the entire system, which should be avoided.

In the phase of implementing a formalized system of business rules as a Web service, again in cooperation with Devika, it would also be desirable if Devika could cooperate with locally installed software. The reasons for this do not need to be explained now.

[erkinalp]

looks similar to #290 (reply in thread)

[franklinselva]

@Okrim007 The scenario you have mentioned is similar to self-improving agents. But why do you think the user needs to intercept every time? Are you proposing this as a starting point?

[Okrim007]

Behind the proposal is the not entirely imperceptible experience of modeling self-referential logical puzzles/quizzes in F-logic with the help of ChatGPT, partly described in the book: https://www.researchgate.net/publication/370801770_Digitalna_humanistika_Neka_videnja (written in Croatian, but Google Translate will easily translate the corresponding passages).

F-Logic is a very advanced logical formalism, which, together with the implementation of Prolog, XSB Prolog and Hilog (a logical programming language with second-order syntax (predicates are also variables) and first-order semantics) is embodied in the ErgoAI development environment.

The ErgoAI environment is ideal for formalizing semi-structured textual data/stories. However, ChatGPT did quite poorly in modeling self-referential puzzles. This means that a large number of interactions with the user (myself), pointing out errors, inconsistencies in logical reasoning, and repeated iterations of the entire process were necessary. Each model provided by ChatGPT had to be "manually" uploaded to the F-logic interpreter, run the interpreter and re-pass error messages to ChatGPT, again "manually". This, of course, is not a problem in the context of modeling toy problems, such as self-referential puzzles, but it becomes very pronounced in the case of the scenario described in the original discussion, where the number of original, textually described business rules amounts to several hundred of them, and in addition, all of them must be shaped into a functional, consistent, non-monotonous "truth maintenance system".

In conclusion, and this follows from my original discussion, I advocate as few "manual" user interactions with Devika as possible, of course, giving Devika full freedom to initiate interactions with the user from her side.

I don't know to what extent Devika's idea fits into that, but for me, the ideal Large Language Model would be one that would unite "sensory" intelligence (through statistical learning through neural networks) and "cognitive" intelligence (through automatic logical deduction), which would include models of heuristic reasoning (Proofs and Refutations, Imre Lakatos), systems of the F-Logic type (ErgoAI), systems such as Coq (Inria, France) and systems of epistemic logic (Reasoning about knowledge, Fagin and all).

One small desirable practical step would certainly be that Devika can render and interpret math formulas written in Latex (so that, for example, it calls (locally installed) KaTex).

In the end, I do not rule out the possibility that this discussion in the context of Devika's development has completely missed what is meant by the team developing Devika, to whom I certainly wish success in Devika's development.