Adala v0.0.4 - Introducing a New Code Environment and Enhanced Automated Prompt Engineering

Major updates

• New Prompt Engineering Algorithm: The upgraded prompt engineering algorithm implements the PE2 method, which significantly enhances the quality of created prompts.
  • Benchmark Update: Demonstrates a considerable advancement over the zero-shot setup, with a 20% improvement demonstrated in the GSM-8k benchmark.
• CodeEnvironment Introduction: This feature allows the execution and validation of code generated by the Agent. For instance, Python code synthesized by the Agent is checked for errors within this environment, enabling continuous improvement.
  • Guide to Code Generation: A tutorial on crafting a code generation agent, focusing on Python script development for data format conversion.
• Custom Matching Function: Users can now define their own matching function to align agent predictions with ground truth examples.

Minor improvements

• instructions_first Parameter: This offers greater control over how the final prompt is structured prior to execution by the Runtime, particularly useful for tasks involving large pieces of text.
• Prompt Diffs: This feature in the Notebook environment highlights incremental changes in the instruction generation history.
• Skill Improvement Without Explicit Feedback: Enables enhancement of skills indirectly.
• CI/CD Enhancements: Integration of code formatting and test-checking features.
• OpenAI API Retry Mechanism: Addition of retries for the OpenAI API, addressing issues reported in #41.

Full Changelog: 0.0.3...0.0.4

Adala v0.0.3 – Extending agent’s feedback and control; core skills update; parallel execution, and more.

Adala 0.0.3 introduces key updates enhancing AI agent feedback and control. Notable features include the extension of feedback collection to prediction methods and new console and web environments for improved feedback integration.

We've decoupled Runtimes from Skills, allowing for more flexible skill definitions and controlled outputs. This update also refines basic skill classes into three core classes: TransformSkill, AnalysisSkill, and SynthesisSkill, enabling better composition and class inheritance. Additionally, ParallelSkillSet allows for concurrent skill execution, increasing efficiency.

The update also improves skills connectivity with multi-input capabilities and introduces a new input/output/instruction format for greater clarity and precision in skill deployment.

Environments

We’ve extended at what point feedback can be collected back into the agent through an environment. Before it was possible only to do that during the learn step, but now it’s possible to request validation while running the prediction method. We also introduce new integrations where feedback could be collected:

• Console environment - asking user to provide feedback via console terminal or jupyter notebook
• Web environment - connecting to 3rd-party API to retrieve user feedback. This is a wrapper to support webhook based communication for services like Discord, Twilio, and alike.
  • Specific feedback protocol follows the following policy:
    • Request feedback POST /request-feedback
    • Retrieve feedback GET /feedback
  • Web server example implementation that uses Discord app to request human feedback

Changes to the collection of user feedback

Adala's latest update enhances its ability to interpret more subtle user feedback. In the previous version, Adala v0.0.2, users needed to provide explicit ground truth labels or texts as feedback. For instance, if the agent predicted "Positive," the user had to respond with the correct label, like "Negative," to indicate a mistake. However, in the new version, Adala v0.0.3, users have the flexibility to give feedback in various forms, such as simple affirmations or rejections (Yes/No, Accept/Reject) or even by guiding the agent with reasoning, without needing to state the explicit ground truth label. For example, if the agent predicts "Positive," a user could respond with "Reject" or explain why the prediction is incorrect.

Runtimes

Runtimes are now decoupled from Skills constrained generation schema. That enables skills to be defined using generic template syntax, and runtimes support controllable output through the use of native methods, or frameworks like guidance.

Skills

Refactoring skills into 3 core parts, that together cover a variety of different ways how data processing can happen:

• TransformSkill – can process dataframe, record by record, and returns another dataframe. Typical use cases - automated data labeling or summarization for the datasets of structured data.
• AnalysisSkill – can compress dataframe into a single record. Use case: data analysis, dataset summarization, prompt finetuning.
• SynthesisSkill – can create a new dataframe based on a single record. For example, synthesizing dataset given the prompt

Combining these skills with LinearSkillSet or ParallelSkillSet (new! from external contribution) enables complex agent workflows.
ParallelSkillSet - combines multiple agent’s skills together to perform data processing in parallel. For example, you may classify texts in multi-label setup, or perform self-consistency checks to improve the quality of LLM-based chatbots.

Skills connectivity

Updates in a way how skills interconnect – multi-input skills. It enables agents to take multiple data columns as input to infer with additional control parameters (e.g. target language, style, or any additional data). For example, it is possible to include two data columns:

TransformSkill(input_template="Translate {text} into {language}", ...)

Skills format

The input/output/instruction template now supports simple f-string specifications and JSON-schema defined variables format. This approach decouples
For example:

TransformSkill(
	instructions="Translate text and classify it"
	output_template="Translation: {translation}, classification: {classes}"
	field_schema={
		"classes": {
			"type": "array",
			"items": {
				"type": "string",
				"enum": ["Positive", "Negative"]
	...
)

Adala v0.0.2 – Introducing Skill Sets!

Focusing on Complex Tasks

We're introducing skill sets, a concept focused on nuanced and sophisticated data processing that reflects the complex nature of real-world problem-solving. In practical scenarios, skills are rarely isolated and often require a multifaceted approach where the output of one skill feeds into the input of another.

By simulating this reality, skill sequencing allows for a granular level of control and customization in automated workflows, facilitating the breakdown of complex tasks into simpler, sequential operations. This not only improves the manageability of data processing tasks, but also enhances the capability of systems to learn and adapt to a variety of scenarios.

Each isolated skill is still verified through the continuous interaction with the environment, improving the overall system's reliability.

Enhancements

• Agents can now learn skills as a skillset (see the linear_skillset_program example). (#21)
• Added fuzzy matching for ground truth comparisons (matching.py). (#21)
• Built out the testing framework (see the tests directory).
• Added the ability to analyze with reasoning (chain of thoughts error analysis).

Bug Fixes

• Fixed an issue where agent creation was failing without GPT4. (#20) Special thanks to @shenxiangzhuang for the first community contribution 🥳

Contributors

New Contributors

• @shenxiangzhuang

Contributors

• @niklub
• @makseq
• @deppp

Adala live!

0.0.1

Without support of Python 3.12 at the moment