RFP Builder Developer Guide

RFP Builder - Developer Guide

User page: Contracts - RFP Builder · pages contracts/rfp-builder/ · endpoints api/rfp-builder/ · helpers includes/rfp_ai.php, includes/rfp_scope.php, includes/rfp_docx_parser.php

In plain words

An RFP starts as a pile of Word documents, one per department, each saying what that department wants from a new system. The RFP Builder reads the text out of each document, asks an AI to list every requirement in it, then asks the AI once more to merge the lists from every department into one set: sorted into categories, given a priority, with contradictions flagged. The analyst tidies that set by hand and locks it. Only then does the AI write the document - an introduction, scope and response instructions, plus one section per category - which the analyst edits and prints. Invited suppliers are then scored 0-5 against every requirement, and a compare page ranks them.

Everything below an RFP belongs to that RFP and follows its company. The AI settings (provider, model, key) are the RFP Builder's own, not the shared AI Providers panel.

Files

File	Job
contracts/rfp-builder/index.php	The RFP list, with New / Edit / Delete (get_rfps, get_rfp, save_rfp, delete_rfp)
contracts/rfp-builder/view.php	One RFP's overview: phase tiles linking to the pages below, and the AI activity panel (get_processing_log)
contracts/rfp-builder/documents.php	Upload, view text, re-extract, run AI extraction on, and delete the source .docx files
contracts/rfp-builder/extracted.php	Every extracted requirement across the RFP's documents, filter by department and type, edit and delete
contracts/rfp-builder/consolidate.php	The consolidation workbench (1,712 lines) - see consolidate.php
contracts/rfp-builder/coverage.php	Category x department heatmap (get_coverage)
contracts/rfp-builder/document.php	The generated document: framing sections, per-category sections, Generate all, TinyMCE edit, version history, restyle
contracts/rfp-builder/preview.php	Print-friendly render of the whole document, built server-side with no API call. PDF is the browser's print
contracts/rfp-builder/suppliers.php	Invite existing suppliers, create a prospective one inline, demo dates and notes
contracts/rfp-builder/scoring.php	Score one supplier at a time, 0-5 per requirement, autosave, radar chart
contracts/rfp-builder/compare.php	Read-only cross-supplier compare: ranked cards, radar overlay, category winners (get_compare_data)
contracts/rfp-builder/audit.php	The full, filterable rfp_processing_log for one RFP
contracts/rfp-builder/help.php	Static in-app help
contracts/rfp-builder/uploads/	Where uploaded .docx files are stored - see Uploads
includes/rfp_ai.php	Every AI call: settings, the HTTP calls, streaming, the prompts, the log
includes/rfp_scope.php	Company scoping: rfpGate(), rfpCanView(), rfpListSql() and friends
includes/rfp_docx_parser.php	rfpExtractDocxText() - plain text out of a .docx
contracts/settings/index.php	The two settings tabs, RFP departments and RFP AI
tests/rfp-companies.php	Fails if an endpoint is added without the company gate - see Company scoping

Every page calls requireModuleAccess('contracts'). Pages fetch from api/rfp-builder/ through a API_BASE constant set to '../../api/rfp-builder/'.

The flow

The code's comments number the AI calls as passes; the user page numbers the phases the analyst sees. Both are below.

1. Create the RFP - save_rfp.php. Status starts at draft. The allowed statuses are draft, collecting, consolidating, generating, scoring, closed, abandoned.
2. Upload (documents.php -> upload_document.php) - .docx only, 20 MB max (RFP_UPLOAD_MAX_BYTES). Each document can carry an rfp_departments id. The file is saved, a rfp_documents row is written with status uploaded, and text extraction runs straight away.
3. Text extraction - rfpExtractDocxText() turns the .docx into flat text into rfp_documents.raw_text, status extracted. A failure leaves the upload in place with status error; extract_document.php re-runs extraction on the stored file.
4. Pass 1 - AI extraction, per document (extract_requirements.php -> rfpAiExtractRequirements()). It deletes that document's earlier extracted rows first, then inserts one rfp_extracted_requirements row per item: text, type (requirement, pain_point or challenge - anything else becomes requirement), source quote, confidence clamped to 0-1. Document status becomes processed. This is a plain request, not a stream.
5. Pass 2 - consolidation, one call for the whole RFP (consolidate.php -> run_consolidation.php -> rfpAiConsolidateStreaming()). The AI returns one JSON object with categories (asked for 8 to 20), consolidated_requirements (each with a priority, critical/high/medium/low, a rationale and the source_extracted_ids it came from) and conflicts. Only after the stream finishes does one transaction delete the RFP's consolidated requirements and categories and write the new ones, the rfp_consolidated_sources links (source ids not in the RFP are ignored) and rfp_conflicts rows with resolution open. Status moves from draft/collecting to consolidating.
6. Hand editing on consolidate.php - edit, add, split, merge, delete, resolve conflicts.
7. Lock - lock_consolidation.php sets is_locked on every consolidated row of the RFP (or clears it) and moves the status to generating (or back to consolidating).
8. Pass 3 - section per category (document.php -> generate_section.php -> rfpAiGenerateSectionStreaming()), HTML into rfp_output_sections.
9. Pass 5 - framing (generate_framing.php -> rfpAiGenerateFramingStreaming()): introduction, scope and response_instructions (RFP_AI_FRAMING_KEYS) into rfp_document_sections. It sends the RFP name, the analyst's context note (rfps.framing_context_text, saved by update_framing_context.php) and the category list - not the requirements.
10. Pass 4 - restyle (restyle_section.php, restyle_framing.php -> rfpAiRestyleStreaming()): re-applies the style guide to existing HTML without changing its meaning.
11. Output - preview.php renders framing then category sections for printing. There is no .docx or PDF writer; "export" is the browser's print.
12. Suppliers and scoring - invite_supplier.php / create_prospective_supplier.php into rfp_invited_suppliers; save_score.php upserts rfp_scores; get_scores.php and get_compare_data.php roll them up.

Steps 8 to 10 refuse while any consolidated requirement of the RFP is unlocked - the check is in each of the four endpoints, answered as a stream error. The two restyle endpoints also refuse when there are no consolidated rows at all; the two generate endpoints do not count that case as unlocked.

How the AI is called

Its own settings, its own two providers

rfpAiGetSettings() reads four system_settings rows: rfp_ai_provider, rfp_ai_api_key (encrypted, listed in includes/encryption.php), rfp_ai_model and rfp_ai_verify_ssl. The provider must be one of RFP_AI_VALID_PROVIDERS = ['anthropic', 'openai']; a missing provider, key or model throws a RuntimeException telling the analyst to set it under Contracts -> Settings -> RFP AI.

This is separate from AiProvider in includes/ai_provider.php. That file's header says its request shapes mirror rfp_ai.php and that it is "intentionally independent so the RFP builder isn't affected". Azure, OpenRouter and OpenAI-compatible servers are not available here. AI Providers says the same from the other side.

Settings are written by save_ai_settings.php, read (key masked) by get_ai_settings.php and tried by test_ai_connection.php, which sends a tiny prompt with its own rfpTestAnthropic() / rfpTestOpenAI(). The same tab saves rfp_default_style_guide.

The functions

Function	What it does
rfpAiCall()	One non-streaming request, Anthropic or OpenAI by setting. Defaults max_tokens 4096, temperature 0.0
rfpAiCallAnthropic() / rfpAiCallOpenAI()	The two request shapes. Anthropic marks the system prompt for caching
rfpAiHttpPostWithRetry()	cURL POST, retries 429, 5xx and network errors up to RFP_AI_RETRY_MAX (3) with backoff from RFP_AI_RETRY_BACKOFF_MS (2000 ms, doubling); timeout RFP_AI_HTTP_TIMEOUT (120 s)
rfpAiCallAnthropicStreaming()	Streaming request to Anthropic. Calls $onEvent('text', {delta}) and $onEvent('usage', {...}) as data arrives. One attempt, no retry - once text has gone to the browser it cannot be replayed. No overall timeout; it gives up after 60 s with no bytes
rfpAiExtractRequirements()	Pass 1, via rfpAiCall(), max_tokens 8192
rfpAiConsolidate() / rfpAiConsolidateStreaming()	Pass 2, max_tokens 16000. Only the streaming one has a caller (run_consolidation.php)
rfpAiGenerateSectionStreaming()	Pass 3. Loads the category's consolidated requirements (ordered by priority) and the departments behind each one through rfp_consolidated_sources
rfpAiGenerateFramingStreaming()	Pass 5, one framing key at a time
rfpAiRestyleStreaming()	Pass 4
rfpAiResolveStyleGuide()	rfps.style_guide, else system_settings.rfp_default_style_guide, else RFP_AI_FALLBACK_STYLE_GUIDE
rfpAiLogAction()	One rfp_processing_log row per call, success or error

Every helper strips a stray ```json fence before json_decode(), and Pass 2 refuses a reply missing any of its three keys.

🔴 Streaming is Anthropic only

rfpAiCallAnthropicStreaming() throws "Streaming is currently only supported with the Anthropic provider." for any other provider. Consolidation, generation, framing and restyle all stream, so with openai saved only Pass 1 extraction (and Test) works - everything from consolidation on stops with that error.

Used outside the RFP Builder

rfpAiCallAnthropicStreaming() takes an optional $settingsOverride so another feature can stream on its own key. api/forms/ai_generate.php, api/knowledge/writeup_stream.php and api/tickets/ai_cleanup_reply.php call it this way, and api/tickets/ai_merge_summary.php requires the file. Changing its signature or event shape changes those features too.

The verify-SSL setting is not used

rfp_ai_verify_ssl is still saved and read, but every cURL call goes through sslApplyCurl(), which follows the global switch alone - see SSL Verification Global Setting. The parameter comments on rfpAiHttpPostWithRetry() and the test functions say so.

The streams

The five streaming endpoints are run_consolidation.php, generate_section.php, generate_framing.php, restyle_section.php and restyle_framing.php. Each:

• is a GET (the browser's EventSource cannot POST), with ids in the query string;
• calls csrfRequireToken() because it writes - assets/js/csrf.js adds ?_csrf= to every EventSource URL (CSRF Protection);
• opens with session_start(['read_and_close' => true]), turns off output buffering, sends X-Accel-Buffering: no for nginx, and set_time_limit(0);
• checks module access with analystCanAccessModule() and the RFP with rfpCanView(), answering both as an error event, because requireModuleAccessJson() or rfpGate() would print JSON into the stream;
• sends phase, text, usage, then complete or error through a local sse() function. generate_section.php can also send skipped.

The pages close the EventSource on complete and on error.

Skip if unchanged

generate_section.php hashes the inputs (category name and description, each requirement's id, text, type, priority and departments, and the style guide) and sends skipped without calling the AI when the hash equals rfp_output_sections.requirements_hash, force is not set and the section is not manually edited. A manually edited section is therefore always regenerated - the old content goes to rfp_section_history first.

⚠️ The endpoint builds that hash itself, with its own copy of the style-guide lookup: the RFP's guide, else the hard-coded fallback. It does not read rfp_default_style_guide, while the stored hash comes from rfpAiGenerateSectionStreaming(), which uses rfpAiResolveStyleGuide(). When an RFP has no guide of its own but a system default is saved, the two hashes never match and the skip never fires. Keep the two in step if either changes.

generate_framing.php reads inputs_hash but its skip block is empty: framing always calls the AI and saves.

Versions

rfp_output_sections.version goes up on every generate, manual edit (update_section.php), restyle and restore (restore_section_version.php), and the previous content is copied into rfp_section_history each time. Manual edits and restores set is_manually_edited = 1; generate and restyle set it to 0. Framing (rfp_document_sections) has no history - a restyle or regenerate overwrites it.

consolidate.php

The biggest page in the module. It loads everything from get_consolidated.php (loadAll(), render()), shows requirements grouped by category with their source quotes (toggleSources()), the AI rationale, and a conflicts panel.

UI	Function	Endpoint
Run consolidation (stream modal)	runConsolidation()	run_consolidation.php
Edit / add custom	openEditModal(), openAddModal(), saveEdit()	update_consolidated.php, add_consolidated.php
Delete	deleteRow()	delete_consolidated.php
Split one into several	openSplitModal(), saveSplit()	split_consolidated.php
Merge selected rows	onSelectRow(), openMergeModal(), saveMerge()	merge_consolidated.php (rfpGate('consolidated', 'ids') checks every id)
Resolve / reopen a conflict	openResolveModal(), saveResolve(), reopenConflict()	resolve_conflict.php
Merge from a conflict	mergeFromConflict()	merge_consolidated.php - the conflict row goes by cascade
Lock / unlock	toggleLock(), applyLockState()	lock_consolidation.php

While the stream runs, updateProgressTracker() counts "name":, "requirement_text": and "consolidated_a_index": in the partial JSON to show live category, requirement and conflict counts.

Split drops any source id not given to a new row, and deletes the original row's conflicts with it. Add custom sets ai_rationale to "Manually added".

⚠️ The lock is enforced in the browser for editing. applyLockState() hides the Run button while locked, but run_consolidation.php and the edit, split, merge and delete endpoints do not check is_locked. Only the four generate and restyle endpoints refuse on it.

Database

All created by Database Verification (includes/db_verify_schema.php) and in database/freeitsm.sql.

Table	Holds
rfps	The RFP: tenant_id, name, status, contract_id, chosen_supplier_id, style_guide, framing_context_text, created_by_analyst_id
rfp_departments	Shared department list (name unique, colour, sort_order, is_active) - Contracts -> Settings
rfp_documents	One upload: department_id, stored filename, original_filename, file_path, raw_text, status
rfp_extracted_requirements	Pass 1 output: document_id, requirement_text, requirement_type, source_quote, ai_confidence, is_consolidated
rfp_categories	Pass 2 categories, sort_order
rfp_consolidated_requirements	Pass 2 output: category_id, text, type, priority, ai_rationale, is_locked
rfp_consolidated_sources	Which extracted rows fed which consolidated row (unique pair)
rfp_conflicts	Two consolidated rows, ai_explanation, resolution, resolution_notes, who and when
rfp_output_sections	One per category: section_content HTML, version, is_manually_edited, requirements_hash
rfp_section_history	Earlier versions of an output section
rfp_document_sections	Framing: section_key (unique per RFP), content, inputs_hash, is_manually_edited
rfp_invited_suppliers	Supplier on an RFP (unique pair), demo_date, notes
rfp_scores	One per RFP, supplier, analyst and consolidated requirement (unique): score (NULL or 0-5), notes
rfp_processing_log	Every AI call - see Audit

Every child table cascades from rfps, so delete_rfp.php deletes one row and then unlinks the files it listed first. Cascades to know about:

• Re-running consolidation deletes the categories and consolidated rows, which cascades to rfp_output_sections (and their history), rfp_conflicts and rfp_scores. Generated sections and supplier scores do not survive a re-run.
• Re-running Pass 1 on a document deletes its extracted rows, which removes their rfp_consolidated_sources links.
• rfp_scores does not cascade from rfp_invited_suppliers, so remove_invited_supplier.php deletes the scores itself.
• Deleting a supplier (ContractsService::deleteSupplier()) clears rfps.chosen_supplier_id and deletes its invitations and scores; deleting a contract clears rfps.contract_id.

is_consolidated on extracted rows is returned by get_extracted.php but nothing in the RFP Builder sets it.

Company scoping

Covered on the user page. In short, from includes/rfp_scope.php:

• rfps.tenant_id is scoped data; NULL is the Default company. Children have no company column - they follow their RFP.
• rfpGate($kind, $key) runs straight after requireModuleAccessJson('contracts') in every JSON endpoint about an RFP. It reads $key from the query, form or JSON body, finds the RFP with rfpIdOf() (kinds rfp, document, extracted, consolidated, conflict, section, framing, invited) and answers {"success":false,"error":"RFP not found"} for one out of reach - never "forbidden", so ids cannot be probed. A list of ids must all pass. On a single-company install, or before tenant_id exists (rfpTenantReady()), it does nothing.
• The streams use rfpCanView(); get_rfps.php uses rfpListSql() (the active company, or every reachable one under All companies); preview.php answers 404.
• save_rfp.php refuses a company the analyst cannot reach and a contract in another company. The Default company is stored as NULL.
• rfp_departments and suppliers are shared by every company. get_rfp_departments.php, the department endpoints and the AI settings endpoints take no RFP and have no gate.
• ContractsService::assertMovable() refuses to move a contract to another company while an RFP of the old company links to it.

A new endpoint must call the gate - php tests/rfp-companies.php fails otherwise. See Multi-Tenancy Developer Guide.

Permissions

The RFP Builder has no capability of its own for using it: Contracts module access is the whole gate for RFPs, documents, requirements, sections, suppliers and scores. Two capabilities in includes/capabilities.php cover the settings tabs only:

Capability	Key	Checked by
Cap::CONTRACTS_RFP_DEPARTMENTS	contracts.rfp_departments	save_rfp_department.php, delete_rfp_department.php, the RFP departments tab
Cap::CONTRACTS_RFP_AI	contracts.rfp_ai	get_ai_settings.php, save_ai_settings.php, test_ai_connection.php, the RFP AI tab

Endpoints check them with requireCapabilityJson(). See Roles Developer Guide.

Scoring is single-blind: get_scores.php returns the analyst's own scores plus other analysts' count and average, never their names or individual scores.

Uploads

Files go to contracts/rfp-builder/uploads/ as rfp<id>_<time>_<random>_<safe name>.docx; rfp_documents.file_path holds the full path on disk. The folder's .htaccess and web.config deny all web access, deploy/nginx/freeitsm.conf returns 404 for it (Running on nginx), and includes/uploads.php lists it in UPLOAD_HTACCESS_ROOTS. Nothing serves the files back - the text is read from raw_text (get_document_text.php).

The .docx reader needs PHP's zip extension

rfpExtractDocxText() opens the file with ZipArchive, reads word/document.xml with DOMDocument, and returns paragraphs outside tables as lines, then each table row as cell | cell | cell. Headings, lists, images, comments, tracked changes, headers and footers are dropped.

Without the zip extension new ZipArchive() is a PHP Error, which catch (Exception) does not catch, so the function checks class_exists('ZipArchive') first and throws a plain RuntimeException saying the extension is needed. upload_document.php and extract_document.php catch it and set the document to error. Read the text from a .docx anywhere else through this function, not ZipArchive directly. Background: Docker Image Missing PHP Extensions.

Audit

rfp_processing_log is the RFP Builder's audit trail, and it records AI calls only: action (extract, consolidate, generate_section, generate_framing, restyle_section), status (success/error), document_id or section_id, tokens_in, tokens_out, and details - JSON with counts, duration, cache figures and model for a success, the error message for a failure. get_processing_log.php feeds the overview's AI activity panel and audit.php.

Hand edits (editing, splitting, merging, scoring, locking) are not logged anywhere; the record of section content is rfp_section_history.

TRAPs

There is one TRAP: comment in these files:

• upload_document.php - upload error messages are an array lookup, not match(). A file the web server runs directly must parse on old PHP, or includes/php_version_guard.php never gets to say "this needs PHP 8.1" - PHP reads the whole file before running any of it.

Not marked TRAP: but worth knowing, all described above: streaming is Anthropic only; rfpAiCallAnthropicStreaming() is shared with Forms, Knowledge and Tickets; re-running consolidation deletes sections and scores; the lock is only enforced on generation; and the section skip hash differs from the stored one when only a system style guide is set.

Related

• Contracts - the user page
• AI Providers - the shared AI settings the RFP Builder does not use
• CSRF Protection - GET streams that write
• Multi-Tenancy Developer Guide
• Code Traps