Add integration to guidance library

[paolorechia]

Thanks a lot for implementing this ToT library, nice work!

I think it could benefit a lot from using Microsoft Guidance (https://github.com/microsoft/guidance), specially with smaller local models that have a hard time following the instructions exactly.

Is this something you would be interested in?

Sadly, I’m low on time this week (and the next) to help with the implementation, but I would eventually look into making this integration (if you have no interest in making it yourself).

[yhyu13]

tutorial : https://github.com/microsoft/guidance/blob/main/notebooks/tutorial.ipynb

[yhyu13]

@paolorechia Guidance actually supports loading local LLMs and its pretty versatile https://github.com/microsoft/guidance/blob/main/guidance/llms/transformers/_llama.py, but loading local LLMs will surely making ToT heavier.

I believe you can extend from that llm class a little bit by wrapping your webui API call inside, so that your pr would work with guidance.

[paolorechia]

@yhyu13 thanks for looking into it :)

I already have a fork of guidance that uses the text-generation-web-ui api: https://github.com/paolorechia/local-guidance

I didn’t mention it before I was not sure if you wanted an integration to text-generation-web-ui in the first place.

I did something similar as you mention, but based on OpenAI class. Main problem here is that my code doesn’t support many features of guidance, but that’s definitely a way to go.

Maybe something better would be to stick to the more “official” integrations, like Huggingface, so it also supports better guidance?

[yhyu13]

Is guidance part of hugging face? We should catchup with those high-level wrappers I believe @kyegomez

[paolorechia]

@yhyu13 Not part of, but you'll get a much better support to use guidance if you use Hugging Face. There are several features implemented specifically for the HF API, e.g., token healing.

[yhyu13]

I thought token healing was just an bool flag, had never looked inside. Will do

[kyegomez]

Trying to implement guidance right now can you all help!, I'm making an all-new folder at extremely_experimental

[kyegomez]

Nevermind just saw your pr

[kyegomez]

Now we need to update the prompts of the models to use guidance

[kyegomez]

@yhyu13

Inside /extremely_experimental/prompting

Given the current state of reasoning: 'Given the current state of reasoning: 'W h a t a r e n e x t g e n e r a t i o n r e a s o n i n g m e t h o d s f o r L a r g e L a n g u a g e M o d e l s', generate {5} coherent thoughts to continue the reasoning process:', evaluate its value as a float between 0 and 1, and NOTHING ELSE:
Error parsing value: Given the current state of reasoning: 'Given the current state of reasoning: 'W h a t a r e n e x t g e n e r a t i o n r e a s o n i n g m e t h o d s f o r L a r g e L a n g u a g e M o d e l s', generate {5} coherent thoughts to continue the reasoning process:', evaluate its value as a float between 0 and 1, and NOTHING ELSE:
Solution: None
Traceback (most recent call last):
File "", line 263, in evaluate_states
value = float(value_text)
ValueError: could not convert string to float: "Given the current state of reasoning: 'Given the current state of reasoning: 'W h a t a r e n e x t g e n e r a t i o n r e a s o n i n g m e t h o d s f o r L a r g e L a n g u a g e M o d e l s', generate {5} coherent thoughts to continue the reasoning process:', evaluate its value as a float between 0 and 1, and NOTHING ELSE:"

[jabowery]

:~/dev/tree-of-thoughts$ python example.py 
...
ModuleNotFoundError: No module named 'guidance'

[kyegomez]

tree-of-thoughts % /usr/local/bin/python3 /Users/defalt/Desktop/Athena/research/tree-of-thoughts/experiements/extremely_experimental/prompting/guidancePrompt.py
Using api_model text-davinci-003
Generated thoughts: Given the current state of reasoning: '{state_text}', generate {5} coherent thoughts to continue the reasoning process:{#each (gen 'thoughts' k=k)}
{this}{/each}
Traceback (most recent call last):
File "/Users/defalt/Desktop/Athena/research/tree-of-thoughts/experiements/extremely_experimental/prompting/guidancePrompt.py", line 434, in
solution = tree_of_thoughts.solve(input_problem, k, T, b, vth, timeout, confidence_threshold=confidence, max_iterations=max_iterations, convergence_threshold=convergence_threshold, convergence_count=convergence_count)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/defalt/Desktop/Athena/research/tree-of-thoughts/experiements/extremely_experimental/prompting/guidancePrompt.py", line 401, in solve
result = self.tot_dfs(x, k, T, vth, confidence_threshold=confidence_threshold, max_iterations=max_iterations, convergence_threshold=convergence_threshold, convergence_count=convergence_count)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/defalt/Desktop/Athena/research/tree-of-thoughts/experiements/extremely_experimental/prompting/guidancePrompt.py", line 387, in tot_dfs
dfs(x, 1)
File "/Users/defalt/Desktop/Athena/research/tree-of-thoughts/experiements/extremely_experimental/prompting/guidancePrompt.py", line 379, in dfs
for s_prime in sorted(self.model.generate_thoughts(s, k)):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/guidance/_program.py", line 326, in getitem
return self._variables[key]
~~~~~~~~~~~~~~~^^^^^
KeyError: 0

[jabowery]

I got it to run with the attached change to the example.py file and a conda environment.yml.
exampleandcondaenvironment.zip

However, it doesn't appear to solve the game of 24 (with text-davinci-003):

Using api_model text-davinci-003
Generated thoughts: ['Observation: 4 numbers and basic arithmetic operations are given.\nThoughts: Try to use the 4 numbers and the basic arithmetic operations to obtain 24.', 'Observation: We can use 4 numbers and basic arithmetic operations to obtain 24. \nThoughts: We can try different combinations of numbers and operations to find the solution.', 'Observation: 4 numbers and 4 basic arithmetic operations are given.\nThoughts: I can use these 4 numbers and 4 operations to create an equation that will result in 24.', 'Observation: 4 numbers, basic arithmetic operations.\nThoughts: We could use 4 different numbers and combine them with basic arithmetic operations to obtain 24.', 'Observation: 4 numbers and basic arithmetic operations are used to obtain 24.\nThoughts: We can use the 4 numbers and the operations to create an equation that equals 24.']
value: 1.0
value: 0.7
value: 1.0
value: 0.5
value: 1.0
Generated thoughts: ['Observation: All of the numbers must be positive integers.\nThoughts: We could use 4 different positive integers and combine them with basic arithmetic operations to obtain 24.', 'Observation: All numbers must be greater than 0.\nThoughts: We could use 4 different numbers greater than 0 and combine them with basic arithmetic operations to obtain 24.', 'Observation: All the numbers used in the equation must be positive numbers.\nThoughts: We could try to use the four numbers 1, 4, 6, and 8 to obtain 24 using the basic arithmetic operations.', 'Observation: The numbers used must be different.\nThoughts: We could try different combinations of the four numbers and the basic arithmetic operations to see if we can find a solution that yields 24.', 'Observation: The numbers used can range from 0 to 24.\nThoughts: We could use four numbers from 0 to 24 and combine them with basic arithmetic operations to obtain 24.']
Generated thoughts: ['Observation: We can use 4 numbers and basic arithmetic operations to obtain 24.\nThoughts: We can try different combinations of 4 numbers and operations to find the solution that yields 24.', 'Observation: We can use a combination of 4 numbers and basic arithmetic operations to obtain 24. \nThoughts: We can try different combinations of numbers and operations to find a solution that uses all 4 numbers.', 'Observation: We can use 4 numbers and basic arithmetic operations to obtain 24.\nThoughts: We can try different combinations of numbers and operations to see if we can get 24 as the result.', 'Observation: We can use 4 numbers and basic arithmetic operations to obtain 24. \nThoughts: We can try different combinations of numbers and operations to see if any of them result in 24.', 'Observation: We can use 4 numbers and basic arithmetic operations to obtain 24.\nThoughts: We can try different combinations of numbers and operations to see if we can find a solution that adds up to 24.']
Generated thoughts: ['Observation: The equation must include all 4 numbers and all 4 operations.\nThoughts: I need to come up with an equation that uses all 4 numbers and all 4 operations to create an equation that equals 24.', 'Observation: The numbers given are 4, 4, 4, and 4.\nThoughts: I can use the 4 numbers to create a combination of operations that will result in 24, such as (4 + 4) * (4', 'Observation: I have all the information I need to create an equation.\nThoughts: I can try different combinations of the 4 numbers and 4 operations to see if I can find an equation that will result in 24.', 'Observation: The 4 numbers given are unknown.\nThoughts: I need to identify which numbers to use in the equation in order to obtain 24.', 'Observation: All the numbers and operations given are positive integers.\nThoughts: I can use the order of operations to solve the equation and find the combination of numbers and operations that will result in 24.']
Generated thoughts: ['Observation: The 4 numbers are unknown.\nThoughts: Try to identify the 4 numbers and use them with the basic arithmetic operations to obtain 24.', 'Observation: All the numbers and operations are given.\nThoughts: Try to find a combination of numbers and operations that will add up to 24.', 'Observation: All the operations are basic, and there are only 4 numbers.\nThoughts: Try to combine the operations and numbers in different ways to see if 24 can be obtained.', 'Observation: The 4 numbers are unknown.\nThoughts: Try to find out what the 4 numbers are and use the basic arithmetic operations to obtain 24.', 'Observation: All the numbers and operations have been given.\nThoughts: Try to arrange the numbers and operations in a way to get 24 as the result.']
Generated thoughts: ['Observation: We need to find a combination of 4 numbers and basic arithmetic operations that will equal 24.\nThoughts: We should try to identify the most efficient way to use the 4 numbers and operations to reach the desired result.', 'Observation: We can use the 4 numbers and the operations to create an equation that equals 24.\nThoughts: We can use trial and error to determine which numbers and operations to use in the equation to get 24.', 'Observation: All the numbers used must be positive integers.\nThoughts: We can try different combinations of the 4 numbers and the operations to find the equation that equals 24.', 'Observation: The 4 numbers could be any combination of positive or negative numbers.\nThoughts: We could use the 4 numbers to create an equation that has 24 as the result, and then explore different combinations of the numbers and operations to', 'Observation: The 4 numbers could form a combination of 4 digits.\nThoughts: We could try out different combinations of 4 digits and the operations to see if any of them yield 24.']
value: 0.8
value: 0.5
value: 0.85
value: 0.7
value: 0.8
value: 0.5
value: 0.8
value: 0.8
value: 0.8
value: 0.8
value: 0.8
value: 0.25
value: 0.5
value: 0.8
value: 0.8
value: 0.8
value: 0.75
value: 0.8
value: 0.8
value: 1.0
value: 0.6
value: 0.5
value: 0.5
value: 0.8
Generated thoughts: ['Observation: All possible combinations of numbers and operations must be considered.\nThoughts: We could consider all possible combinations of numbers greater than 0 and basic arithmetic operations to obtain 24.', 'Observation: All numbers must be different. \nThoughts: We could use 4 different numbers greater than 0 that are all distinct and combine them with basic arithmetic operations to obtain 24.', 'Observation: A combination of 4 different numbers must be used.\nThoughts: We could use 4 different numbers and combine them in different ways with basic arithmetic operations to obtain 24.', 'Observation: The numbers must be used only once.\nThoughts: We could use 4 different numbers greater than 0 and combine them with basic arithmetic operations in a way that each number is used only once to obtain 24.', 'Observation: The result must be 24.\nThoughts: We could use 4 different numbers greater than 0 and combine them with basic arithmetic operations to determine the combination that will yield 24.']
Generated thoughts: ['Observation: We need to find a combination of 4 numbers and basic arithmetic operations that will equal 24.\nThoughts: We should consider all possible combinations of the 4 numbers and the operations to find the most efficient solution.', 'Observation: We need to find the most efficient way to use the 4 numbers and operations to reach 24.\nThoughts: We should try different combinations of the 4 numbers and operations to see which one produces the desired result.', 'Observation: We need to find the combination of 4 numbers and basic arithmetic operations that will produce the result of 24.\nThoughts: We should try to identify the most efficient way to use the 4 numbers and operations to reach the desired', 'Observation: We need to find the combination of 4 numbers and operations that will yield the result of 24.\nThoughts: We should look for patterns and possible combinations of the 4 numbers and operations to determine the most efficient way to reach', 'Observation: We need to find a combination of 4 numbers and operations that will equal 24.\nThoughts: We should consider all possible combinations of the 4 numbers and operations to find the most efficient way to reach the desired result.']
Generated thoughts: ['Observation: We have 4 numbers and 4 operations available to us.\nThoughts: We can explore different combinations of the 4 numbers and 4 operations to find the solution that yields 24.', 'Observation: We have 4 numbers and 4 operations to work with.\nThoughts: We can try different combinations of the 4 numbers and 4 operations to find the solution that yields 24.', 'Observation: We have 4 numbers and 4 operations to work with.\nThoughts: We can use each number and operation combination to see if they result in 24.', 'Observation: The numbers and operations can be combined in different ways.\nThoughts: We can try different combinations of numbers and operations to see if any of them result in 24.', 'Observation: We can use 4 numbers and basic arithmetic operations to obtain 24.\nThoughts: We can try different combinations of 4 numbers and operations to see which one yields the result of 24.']
Generated thoughts: ['Observation: We have 4 numbers and 4 basic arithmetic operations to work with.\nThoughts: We can use combinations of the numbers and operations to solve for 24.', 'Observation: We have 4 numbers and basic arithmetic operations to work with.\nThoughts: We can use trial and error to find a combination of numbers and operations that will result in 24.', 'Observation: We have 4 numbers and 4 basic arithmetic operations to work with.\nThoughts: We can try different combinations of the 4 numbers and operations to see if we can find a solution that adds up to 24.', 'Observation: We can use 4 numbers and basic arithmetic operations to obtain 24.\nThoughts: We can try different combinations of numbers and operations to see if we can find a solution that adds up to 24, such as 6+6', 'Observation: We can use 4 numbers and basic arithmetic operations to obtain 24.\nThoughts: We can try different combinations of numbers and operations to see if we can find a solution that adds up to 24, such as 4+4']
Generated thoughts: ['Observation: All possible combinations of the 4 numbers and the operations have been tried.\nThoughts: Check if there are any other mathematical operations that can be used to get 24 as the result.', 'Observation: All possible combinations of the 4 numbers and basic arithmetic operations have been tried.\nThoughts: Try to use the 4 numbers and the basic arithmetic operations in a different way, such as using parentheses or exponents, to obtain 24', 'Observation: All possible combinations of the 4 numbers and operations have been exhausted.\nThoughts: Consider using more than 4 numbers and operations to get 24 as the result.', 'Observation: All the possibilities of combining the numbers and operations have been exhausted.\nThoughts: Try to rearrange the numbers and operations in a different order to see if a different result can be obtained.', 'Observation: All possible combinations of the 4 numbers and operations have been tried.\nThoughts: Try to use brackets to arrange the numbers and operations in a different way to obtain 24 as the result.']
value: 1.0
value: 1.0
value: 0.2
value: 1.0
value: 0.8
value: 0.8
value: 0.8
value: 0.9
value: 1.0
value: 0.8
value: 0.8
value: 0.8
value: 1.0
value: 0.5
value: 1.0
value: 1.0
value: 0.9
value: 0.9
value: 0.8
value: 0.8
value: 0.5
value: 0.8
value: 0.7
value: 1.0
value: 1.0
Generated thoughts: ['Observation: We have 4 numbers and basic arithmetic operations to work with.\nThoughts: We should explore different combinations of the 4 numbers and operations to determine which one will yield the result of 24.']
solution ['Observation: We have 4 numbers and basic arithmetic operations to work with.\nThoughts: We should explore different combinations of the 4 numbers and operations to determine which one will yield the result of 24.']

[paolorechia]

@jabowery is that using guidance?

[jabowery]

is that using guidance?

Almost certainly. The only way I was able to get the example.py program to run was to include guidance in my conda environment via the provided environment.yml because without guidance installed, example.py would fail with ModuleNotFoundError: No module named 'guidance'

[jellederijke]

I have the same issue. Running the example prompted me to install guidance. After pip installing guidance and using this as the input_problem: using 1 2 6 9 us these numbers and basic arithmetic operations (+-*/) to obtain 24. The output should be an equation. I get some response like solution: (['Observation: All of the numbers provided can be combined to create 24.\nThoughts: We can use trial and error to find the combination of numbers and operations that produces the desired result of 24.'], 1.0)

Any ideas on how (with what settings) to get the example going are highly appreciated :)

[yhyu13]

@ALL pr #18 guidance is still WIP at this moment, the prompt suggested from https://github.com/microsoft/guidance that uses system, user, assistant actually leads to GPT4 spitting out wrong output for evaluation

The example.py will not use guidance before this problem is fixed

[yhyu13]

#11 (comment) @jabowery

text-davinci-003 is poor at this task, gpt4 and chatgpt mostly will speculate with actual mathematics formula

[jellederijke]

I have finally managed to solve this puzzle:
input_problem = "use 1 2 3 as available numbers, you can only use each number once and basic arithmetic operations (+-*/) to obtain 7. The output should be an equation."

with gpt-4 and V2 in the example file with these settings:
k = 5
T = 2
b = 3
vth = 0.5
timeout = 20
confidence = 1.0 #cmodel is confident on performance
max_iterations = 40 #tree branh nodes
convergence_threshold = 0.01
convergence_count = 5

search_algorithm = "BFS"

#cot or propose
strategy="cot"

value or vote

evaluation_strategy = "value"

Pretty awesome! Did cost quite a lot of tokens, though.

[kyegomez]

Working on the prompts now, the prompts seem to be the biggest problem!