How to use Config Studio

How to Use Config Studio

Overview

Config Studio is a guided configuration authoring tool for Interaction History based reporting. It lets you start from a representative IH sample, define preprocessing logic, approve the field catalog, generate a draft configuration with AI, and then review metrics, reports, and chat settings before exporting the final TOML.

Config Studio is intended to replace the older one-shot config generator workflow with a safer, more reviewable process.

Where To Find It

Open the application and navigate to the Config Studio page.

Recommended Workflow

Use the steps in order:

1. Sample
2. Required Fields
3. Defaults
4. Filters
5. Calculations
6. Approve Fields
7. AI Draft Review
8. Metrics
9. AI Report Refresh
10. Report Review
11. Chat with Data
12. App Settings
13. Save & Export

The tool is designed so that each later step depends on the decisions made earlier.

Sidebar Controls

Max Sample Rows

If the uploaded IH sample contains more rows than this limit, Config Studio keeps a random sample of that size and uses it for profiling, preprocessing, field approval, and AI drafting.

This is useful for:

• speeding up schema profiling
• keeping preprocessing responsive
• avoiding AI prompts based on very large raw samples

LLM Settings

The sidebar also contains optional AI settings:

• API key
• model
• reasoning effort
• verbosity

These are only required when you reach AI Draft Review, AI Report Refresh, or AI Repair.

Step-by-Step Guide

1. Sample

Upload one representative IH sample file.

Supported file inputs for the studio:

• parquet
• json
• zip
• gzip

This step shows:

• detected file type and file pattern
• raw schema preview
• raw sample rows

Use a sample that is structurally representative of the production files you want to analyze.

2. Required Fields

Choose the source columns used as:

• OutcomeTime
• DecisionTime

These are then used to derive:

• Day
• Month
• Year
• Quarter
• ResponseTime

Those derived fields appear in later steps and can be used in metrics, reports, and AI suggestions.

Required Interaction History fields are locked into the approved catalog because later preprocessing, metrics, and reports depend on them.

3. Defaults

Defaults define what happens when required or useful columns are missing or null.

Use this step to:

• fill null values in existing columns
• create missing columns with literal defaults

Defaults are applied before filters.

4. Filters

Define a dataset-level Interaction History filter using either:

• simple rule rows
• raw Polars expression

The filter is written into:

[ih.extensions]
filter = """..."""

Config Studio applies filters before calculated fields. If preprocessing produces zero rows, later steps are blocked and the studio asks you to return to the filter, defaults, runtime, or calculated-field settings.

5. Calculations

Use this table to define new columns from Polars expressions.

Examples:

pl.when(pl.col("CustomerID").str.slice(0, 1) == "C").then(pl.lit("Customers known")).otherwise(pl.lit("Device/Anonymous"))

If the expression does not contain .alias(...), Config Studio will automatically alias it to the field name from the Name column.

Calculated fields are written into:

[ih.extensions]
columns = """[...]"""

Subject ID handling can also add a SubjectID calculated field when a non-standard source column is selected.

IH Runtime Settings

Runtime settings are surfaced in the early preprocessing steps because they affect how the runtime reads Interaction History files:

• file_type
• file_pattern
• ih_group_pattern
• hive_partitioning
• streaming
• background

Notes

• file_type Runtime reader selection. In this app the supported values are parquet and pega_ds_export.
• file_pattern Glob pattern used to discover IH files.
• ih_group_pattern Regex used to derive IH file groups from file names.
• hive_partitioning For parquet scans, infer statistics and schema from hive-partitioned paths and use them to prune reads.
• streaming Prefer Polars streaming engine during collection instead of the default auto engine.
• background Run collection in the background and return a handle that can later fetch or cancel the result. Polars marks background mode as unstable.

Runtime Order

Config Studio follows the same broad preprocessing order as the runtime pipeline:

1. apply default values
2. apply IH filter
3. derive time fields
4. add calculated fields

Empty Result Warning

If preprocessing produces zero rows, Config Studio blocks later steps and asks you to return here and relax the filter or adjust preprocessing rules.

6. Approve Fields

This step defines the field catalog that AI is allowed to use.

Required IH Fields

Fields required by the application are locked in automatically and cannot be removed.

Optional Fields

All other available fields can be included or excluded from the approved catalog.

Only approved fields are used later for:

• metric group_by
• metric global_filters
• metric field references
• AI-generated reports and mappings

The field catalog includes automatic tags such as required fields, time fields, likely IDs, business dimensions, numeric measures, high-cardinality fields, and fields needing review. Use those tags to decide which columns should be available to AI.

AI Privacy Summary

Before generation, the studio shows what will be sent to AI:

• approved field names
• fields hidden from AI
• sample values shared with AI
• sample values masked from AI
• approximate prompt size

Uncheck Share Sample Values for fields where the column name is enough and examples should not be included in the prompt.

Use this step to intentionally narrow the configuration surface before AI suggestions are generated.

7. AI Draft Review

At this step the tool combines:

• the approved field catalog
• the current IH preprocessing configuration
• the config template

and asks the LLM to generate:

• [metrics]
• [reports]
• [variants]

The page also shows:

• IH Config Preview
• Approved Schema Preview
• AI Privacy Summary
• the full AI prompt in a popover

If AI generation succeeds, the result becomes a pending draft. It is not immediately applied.

Review the draft changes before accepting:

• added, changed, removed, and unchanged metric sections
• added, changed, removed, and unchanged report sections
• variant changes
• fields and scores referenced by the generated output

You can keep or reject generated metrics and reports. Reports tied to rejected metrics are excluded automatically. Only after Accept Selected Draft does the draft become editable in later steps.

8. Metrics

This step is a review and correction step for AI-generated metrics.

What You Can Edit

• metrics.global_filters
• each metric group_by
• response settings like positive_model_response and negative_model_response
• descriptive columns
• metric-specific field mappings
• metric filters

Important Constraints

Only approved fields can be used for:

• global_filters
• group_by
• field reference settings such as experiment_name, customer_id_col, purchase_date_col, and similar fields

Metric filter expressions are allowed as raw Polars expressions, but Config Studio checks pl.col("...") references and blocks unapproved fields.

After metric edits, continue to AI Report Refresh so reports can be regenerated from the updated metric grouping fields.

9. AI Report Refresh

This step rebuilds only [reports] from the current draft metrics and approved schema.

Use it after metric edits to:

• remove reports that no longer match the metric definitions
• add reports for newly available grouping fields
• keep report keys stable where possible
• rebuild report mappings without changing preprocessing, metrics, variants, UX, or chat settings

The current metrics and current reports are shown side by side, and the report refresh prompt is available in a popover.

10. Report Review

This step reuses the visual report builder.

Use it to:

• delete weak AI-generated reports
• refine field mappings
• switch supported reports between visual and raw modes
• create additional reports manually

The builder remains backward compatible with the current TOML report structure.

The page also shows a report validation summary with:

• total report count
• ready reports
• reports with warnings
• reports with errors
• a per-report table with metric, type, status, and issue summary

AI Repair

When validation finds blocking errors in AI-owned sections, the studio can generate a targeted AI repair.

AI Repair can update:

• [metrics]
• [reports]
• [variants]

It cannot update:

• [ih]
• [holdings]
• [ux]
• [copyright]
• [chat_with_data]

Repair proposals follow the same review-first process as the initial draft:

1. The repair prompt includes the current draft and validation errors.
2. The AI response must include replacement [metrics] and [reports] sections.
3. The proposed repair is stored as pending.
4. The studio shows metric, report, and variant changes.
5. The repair is validated before it can be accepted.
6. Stale repairs are blocked if the draft changed after repair generation.

11. Chat with Data

Use this step to review and adjust:

• ux.chat_with_data
• [chat_with_data]
• [chat_with_data.metric_descriptions]

This is the final place to align the assistant prompt and metric descriptions with the generated metrics and reports.

12. App Settings

Use this step to review app-level settings that are not part of metric or report generation:

• UX settings
• branding
• variant metadata

Chat enablement stays in Chat with Data; this step handles the remaining application settings.

13. Save & Export

This step shows the final combined TOML and enables:

• download of the generated config
• activation of the draft config in the current app session

Config Studio does not expose apply/download earlier in the flow. The goal is to make sure preprocessing, metrics, reports, chat settings, and app settings have all been reviewed first.

Apply and download are disabled while blocking validation errors remain.

Config Health and Review Progress

Config Studio shows two shared review aids near the top of the page:

• Config Health summarizes current validation errors, warnings, and info messages.
• Review Progress shows badges for preprocessing, field approval, AI draft, metrics, reports, AI repair, and final export.

Badge meanings:

• Ready: no issues found for that area
• Needs Review: warnings or pending AI review remain
• Blocked: blocking validation errors remain
• Pending: the area is not available yet

Practical Tips

• Start with a sample that has realistic values, not only schema-complete records.
• Keep preprocessing simple first, then add calculated fields incrementally.
• Use Approve Fields aggressively to remove noisy technical columns before AI generation.
• Review Metrics before AI Report Refresh; most report issues come from weak metric definitions, not from the report builder itself.
• If AI produces too many reports, reject them during AI Draft Review or delete them in Report Review.
• Use AI Repair for blocking metric/report validation errors, but still inspect the repair diff before accepting it.

Limitations

• Config Studio currently starts from IH samples only.
• AI output is constrained, but still requires review.
• Runtime preview of actual charts is not part of the current flow.
• background behavior follows current Polars semantics and should be treated carefully.

Troubleshooting

Preprocessing returns zero rows

Go back to 3. Defaults, 4. Filters, or 5. Calculations and check:

• overly strict IH filter
• incorrect default values
• calculated fields that depend on missing columns

Required fields cannot be removed in Approve Fields

This is expected. Required IH fields are locked because later metrics and reports depend on them.

A metric filter is rejected in Metrics

Check the pl.col("...") references in the filter expression. Every referenced field must be part of the approved field catalog.

AI Draft Review is not available

Make sure:

• preprocessing does not fail
• preprocessing does not return zero rows
• an API key is configured in the sidebar

AI Repair cannot be accepted

Check whether:

• the proposed repair still has metric or report validation errors
• the draft changed after the repair was generated
• the error belongs to a non-AI-owned section such as [ih], [ux], or [chat_with_data]

Discard stale repairs and generate a fresh repair after manual edits.

Result

The final output is a standard TOML configuration file that remains manually editable and compatible with the rest of the application.