Update Conversational Chat output_parser.py to support more final response cases

[treppers]

In heavily utilizing the conversational chat agent, I found that it didn't account for the many potential outputs that the agent could respond with.

The use-cases this now supports are:

• Instances when it doesn't include the json after the markdown escape
• Instances when the response includes special characters aren't JSON escaped
• Instances when the response runs out of tokens and which then results in a malformed JSON response

Fixes # 2567
Fixes # 3448

Community members can review the PR once tests pass. Tag maintainers/contributors who might be interested:

@hwchase17
@mbchang

[armen-tractatus]

It's considered bad practice to 'catch the universal Exception. JSON decoder errors will typically return a ValueError, which includes the JSONDecodeError child class.

[hwchase17]

lets add test cases for this

[treppers]

good idea. where's the best place to put them?

[treppers]

@hwchase17 added a test for output_parser, but wasn't sure where to put it so for now just dumped it into the same directory. please let me know where to move it.

i merged with the latest release on master and also added a specific exception for JSON decoding.

it is slightly opinionated because it assumes the developer wants to see an output if tokens get exhausted or if the completion responds without any markdown which in my experience only happens when it is the final answer.

[baskaryan]

stale, @treppers feel free to ping if you return to it