server/public_simplechat alternate web client ui update - uncompressed 300KB - built in client side tool calls with 0 setup - reasoning - vision - ai calling into itself #17415
+5,439
−506
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Use same to set a placeholder for Authorization entry in headers
Take care of the possibility of content not being there as well as take care of retrieving the tool calls for further processing. With this tool calls should work in non streaming mode also
NOTE: not a robust check, just tries to establish a http connection for now and doesnt really check if it is the specific proxy srvr of interest or not.
Also fetch info from ai-server, and place path and ctx size into current Me instance and include in show info.
Also create a top level div wrt whole. And allow class to be specified for the same as well as the top level legend, optionally
Also ensure when switching between sessions, the full set of props info is shown.
Also update the readme wrt same and related
Part 1 - add el prefix wrt the element handle related vars
Maintain the current property hierarchy to its root over recursive calls. Allow callers to specify the props to be trapped using the prop hierarchy. Pass the prop hierarchy to the fTrapper. This should allow one to trap any prop wrt its editing ui setup, irrespective of whether it is a prop of the main object passed, or a member of a child prop of the main object passed or so ... Update the setting up of ChatHistoryInCtxt and ApiEndPoint to follow the new semantic/flow.
Fix up the oversights wrt any depth trapping flow Remember to start the propWithTree being checked/trapped with : to indicate the root of the prop hierarchy and also use : as sep between the elements of the props hierarchy tree Also had forgotten about the goof up possible with using in in a condition statement to check for array to contain a entry of interest in JS, fixed it now.
Update Me class Update show settings Update show props info Update readme
Moved into Me.chatProps
Also remove more inner/detailed stuff from show info in not bAll mode, given that many of the previous differentiated stuff have been moved into chatProps and inturn shown for now
Dont allow tool names to be changed in settings page
The config entries should be named same as their equivalent cmdline argument entries but without the -- prefix
Allow fetching from only specified allowed.domains
Had confused between js and python wrt accessing dictionary contents and its consequence on non existent key. Fixed it. Use different error ids to distinguish between failure in common urlreq and the specific urltext and urlraw helpers.
with allowed domains set to few sites in general to show its use this includes some sites which allow search to be carried out through them as well as provide news aggregation
Also had forgotten to rename to image_urls, when switching from single image with its corresponding field image_url to image_urls, wrt removing the same after it has been added to the content array.
This was referenced Nov 20, 2025
Contributor
Author
|
Next will be adding tool call to allow ai to use itself in a fresh chat context by letting ai create its own suitable system prompt and chat message as needed, based on its interaction with user and in turn what it is trying to achieve. |
Use variables in css for colors. Change the tool response color to be bit more soothing and kind of syncd with the user collor. However the tool call color is left bit odder, to ensure user sight goes there for validating and triggering.
Instead use the explicit positioning, which was already added to support firefox, given that firefox still doesnt support anchor based positioning. Also move the popover to top of the chat message bubble/block, instead of the bottom area. This ensures that the popover isnt butting heads with the tool call trigger or user input blocks.
Had a custom struct for parts of Me needed by SimpleChat, and inturn that need to be controlled when starting a independent parallel ai session. For now if chat streaming is used or not and tools is enabled or not is allowed to be explicitly controlled when handle_chat_hs is called. Inturn when toolai triggers the parallel ai session, it disables tools calling support wrt this child/external ai tool call. Else even the external ai session may end up triggering tool call wrt external ai for the same job and thus get into a infinite recursive loop. Also * update the toolai meta data a bit. * increase time allowed for toolcall to return a response, especially usefull for external_ai tool call. Cleanup some old notes and update readme in general.
This will help ensure that switching sessions etal wont lead to loosing handle to sessions that one exited from, with additional logic to be added later. Rather I am thinking, maybe it may make more sense to dynamically pick the currently active chat sessions latestResponse field to inturn show into a div element at end of the elDivChat. On thinking further, maybe making elDivStream into elDivStreams with seperate div for each session is the better thing, as the SC.handle_response will be able to update its corresponding div and the session switch can link the currently visible session's div to be mapped to elDivChat. NOTE: Rather this commit in itself should take care of the current normal flow, which wont allow user to switch sessions when a tool call or handshake with server is pending. However in future if I want the ui to be freely switched independent of if a tool call or server handshake is active, then I need the logic mentioned in previous para.
For reasons mentioned in last commit Also include removing and reappending the chat session specific div in uirefresh.
Sometimes the session specific DivStream seems to be missing from DivChat. Explicitly spliting the remove and append just for the heck of it didnt help. So also have used try-catch-finally to ensure it gets added back even if its not already in there, by not allowing removeChild failure to bypass append etal. Retaining this here just for the heck of it. WIll be switching to the appendChild only flow (given it automatically moves the node, as needed).
Missing DivStream caught. Logic ready for a distant future, where one may allow the chat session to be switched even if there is a pending ai server / tool call response. Avoids the unneeded removeChild before calling appendChild. Also found the reason why sometimes the DivStream was missing from DivChat. Rather when I switched from the transient <p> element to the persistant session specific <div> elements, I had forgotten to convert the flow fully, rather I had forgotten to replace the elP.remove() with elP.replaceChildren Also retaining that debug log wrt missing DivStream path, Just in case for now, to cross check once later, I havent missed any other path.
Make DivStream hold a Role element and Data element and inturn have the live got data go into the data element. Set some of the relavent classes wrt these, so that it is themed matching other chat blocks, to any extent. Add a clear helper function to cleanup as and when needed. NOTE: Remember to use this to get hold of the DivStream instance being worked on. NOTE: Dont forget that setting a parent wrt a HTMLElement wont automatically add it to the corresponding DOM with a parent child relation. The new html element will just remain in memory ignored by everyone else.
Add logic for hiding and showing and use them as needed.
Include the DivStream of ExternalAi toolcall in the other chat session UIs, so that user can see what the external_ai toolcall is doing, without having to switch out to external ai session tab. Update the name of the tool call external ai session. Ensure to clear previous chat session wrt external ai tool calls As the ai may call external ai toolcall with the same system prompt sometimes, which wont trigger the autoclear logic, wrt the corresponding chat session. TODO: In future maybe provide a option to continue any previous chat session if the system prompt is not changed wrt external ai toolcall.
Update the external ai tool call description to indicate that it doesnt have access to internet or tool calls. Update the sys_date_time description and avoid the confusion caused to some ai wrt the template string as to whether it is optional or required and wasting reasoning time around it. Now simply state that it is a reqd argument, and suggest the internal default template as a useful one to use. Update the msg returned by data store tool calls, to make them less verbose (ie avoid duplicating key list or got key data) while also more informative (ie num of keys, data length) Update the readme.
Contributor
Author
|
The external_ai tool call implemented, for now this will be calling into a clean slate version of the currently running ai model itself without toolcall or internet access, while allowing custom system prompt and user message of the ai model's choosing. Can help ai with targetted summarising | structured data generation | critic etal, as it deems fit. NOTE: Any auto check failures noted below other than the python check are unrelated to this PR. And the python check is more a factor of the default python related auto check not allowing better type hinting and some other python features, which are already standard in python for last 2 odd years. NOTE: Any WebUI failure wrt auto checks, is more related to the default webui client related failure and not wrt this alternate webui at tools/server/public_simplechat. |
hanishkvc
changed the title
Contributor
Author
|
Look at the latest PR #17451 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi all
In addition to the tool calling, reasoning and vision features added in the previous PRs in this series, this version rebases to the latest master branch, also cleans up the ui a bit freeing up vertical space and adds support for multiple images as part of a single message to be sent to the ai model.
Hi @ggerganov
I hadnt noticed your response few weeks back to my previous older PR in this series, as I was in the process of adding missing features wrt this alternate simple minded web client ui ie peek at reasoning, vision and tool calling and some associated cleanup, so that I could explore some stuff which I wanted to.
If you look at this PR, you will notice that this alternate pure html + css + js based flow (avoids dependence on external / 3rd party libraries in general) now supports reasoning, vision and also tool calling (with a bunch of built in client side based tool calls with zero setup, ++) all within a uncompressed source code size of 300KB (including the python simpleproxy.py for web access and related tool calls). Also the logical ui elements have their own unique id/class, which can help theme the ui, if one wants to.
While the default web ui is around 1.2 MB or so compressed, needs one to understand svelte framework and also track the bundled modules. Also it doesnt support tool calling currently, and the plan is more towards server side / back end MCP based tool call support, if I understand correctly.
Given the above significant differences, it may make sense to continue this as a lightweight alternate ui option within llama.cpp itself, parallel to the default webui. My embedded background also biases me toward simple, flexible and functional options.
NOTE: When I revisited ai after almost a year++ wanting to explore some of the recent ai developments, I couldnt find any sensible zero or minimal setup tool calling supported open source ai clients, so I started on this series of patches/PRs.
Eitherway the final decision is up to you and team of open source developers who work on this proactively, rather than once in a bluemoon me, as to whether you would prefer to apply these into llama.cpp itself or not. Do let me know your thoughts.
NOTE: I am duplicating the response I posted 2 odd days ago to your feedback (which I had missed) to that much older PR, again here, given that lot of features have been added wrt that PR.
NOTE: The immidiate previous PR in this series was #17142