gpt version, unreliable gpt output, parsing issues

[mwesterhof]

When trying to generate some things, I ran into several issues. engineer is currently targeting gpt model 4.0, which doesn't appear to be available. as mentioned in another issue, gpt-3.5-turbo seems to work. After making this change, i noticed that the parsing of the gpt response seemed to have some issues.

It's possible that these are related to me using a different model, but I can't test it with 4.0 as this doesn't seem to work at all. Engineer can't successfully parse the response to extract the correct filenames. I managed to update the regex to fit the output.

after making this change, i noticed that GPT has extremely inconsistent output, and will switch between several different formats, seemingly at random. The only fix i could think of for this was to add some format specifiers in the prompt, but that does seem to work.

this commit seems to solve the issues, but before making this a PR, some discussion is probably needed. Is this even the right way to go about fixing this?

[jebarpg]

@mwesterhof also check this post out #35 (comment)
a couple of posts above that I actual address the same thing you are speaking about here with the formatting and agree that it needs to be corrected on the output of gpt side and not adapted to by a regex side.

[mwesterhof]

@mwesterhof also check this post out #35 (comment) a couple of posts above that I actual address the same thing you are speaking about here with the formatting and agree that it needs to be corrected on the output of gpt side and not adapted to by a regex side.

Indeed. I think that the regex can be somewhat improved anyway (as I tried to do).

It's not just that GPT's output doesn't match the regex, it's that GPT appears to be all over the place regarding how it structures the output. Actually specifying the output formatting as part of the GPT query seems to be a solution. I'm not sure if it's the best solution, but we would need some way to have reliable output

[jebarpg]

@mwesterhof also check this post out #35 (comment) a couple of posts above that I actual address the same thing you are speaking about here with the formatting and agree that it needs to be corrected on the output of gpt side and not adapted to by a regex side.

Indeed. I think that the regex can be somewhat improved anyway (as I tried to do).

It's not just that GPT's output doesn't match the regex, it's that GPT appears to be all over the place regarding how it structures the output. Actually specifying the output formatting as part of the GPT query seems to be a solution. I'm not sure if it's the best solution, but we would need some way to have reliable output

It looks like we have found a good solution which will work for a variety of formats now. Lot more generic in it's regex capturing. We are waiting for this afternoon to create a PR to do some further testing on it but it looks very robust from my analysis.

[jebarpg]

@mwesterhof #35 (comment)

[patillacode]

A lot of progress has been made since this was posted.

If you wish to reopen the issue please do following the new issue template.