Proposal: Tasking Manager Chat Assistant

[prabinoid]

Tasking Manager Chat Assistant

Goal

A chat assistant that answers Tasking Manager questions in natural language, for every role: mappers, validators, project managers, and admins. It works as a Retrieval Augmented Generation (RAG) assistant: answers are grounded in a curated knowledge base, and if the knowledge base doesn't cover something, the assistant should say it doesn't know rather than guess. It should also decline malicious or out of scope requests.

Knowledge base

A corpus built from multiple sources:

• Tasking Manager docs
• Tasking Manager FAQ corpus
• LearnOSM Tasking Manager guides
• OSM wiki pages
• Frontend UI strings and backend API docstrings and error messages

Tasking Manager docs, the LearnOSM guides, OSM wiki pages, and the FAQ corpus are already written in natural language, so they should index cleanly. Frontend UI strings and backend docstrings and error messages are also being included, but they're a less reliable source. They're terse fragments written for a different purpose, like labeling a button or documenting a function, rather than composed as an answer to a question. Indexed raw, they may retrieve on keyword overlap but read badly. They likely need a rewrite or enrichment pass, for example expanding "Task is locked for mapping" into a fuller sentence with context, before embedding.

Retrieval and orchestration

• Orchestration: LlamaIndex
• Vector store: PostgreSQL with pgvector, so we're not introducing a new datastore to run
• Hybrid retrieval: pgvector semantic search combined with Postgres full text search, since Tasking Manager specific keywords and maybe acronyms like "AOI" or specific task states need literal keyword hits that embeddings can miss. We're also evaluating BM25 against plain full text search to see which performs better.

Guardrails and evaluation

• Refuse queries outside the assistant's scope, or that look malicious or unsafe.
• Decline to answer when retrieval confidence is low, rather than fabricating a response.
• Validate with an eval suite of realistic per-role questions, plus prompt injection and other adversarial probes.

Challenge

The assistant is only as good as the knowledge base and its ability to retrieve the right information. The hardest part is deciding what end users actually need. A mapper needs to know how to filter tasks to map, not how authentication works. On top of that, creating the knowledge base and keeping it updated and in sync is its own ongoing effort, since the sources span multiple repos with different release cycles, so docs, UI strings, and backend messages can drift out of step with each other.

Open questions

• What should be included in the knowledge base, and what should be left out?
• Should the chatbot restrict its answers based on the user's Tasking Manager role?
• What should be the LLM and embedding model?
• Should the chatbot support conversation memory and follow up questions?
• What chunking and metadata strategy should back retrieval, for example tagging chunks by doc type, source, role, and freshness? This could also feed a future role based ranking system.

---

Credits to @ajitstha42 for drafting the discussion and tagging along for the implementation of the feature.

[ramyaragupathy]

Some questions I thinkt he chatbot should answer:

• What's my mapper level, and what do I need to reach the next one?
• How many hours have I contributed to Tasking Manager?
• What task do I currently have locked?
• What badges have I earned?
• How is my contribution this month compared to last month?
• What are the projects I have contributed to?
• What's my current mapping streak?
• What's my validation-vs-mapping split?
• What teams am I part of?
• How many projects are there on Tasking Manager?
• How many health-oriented projects are there?
• How many users are registered on Tasking Manager?
• How many tasks have been mapped/validated in total?
• Which countries or organizations have the most active projects?
• What are the most active/trending projects right now?
• Which projects are beginner-friendly / match my skill level?
• Which projects are expiring soon or short on mappers?
• Which projects are in a specific country, org, or campaign?
• What's project X's difficulty level and required editor (iD/JOSM)?
• What's the status of project X (% mapped, % validated)?
• Recommend a project based on what I usually map
• Why was my last task invalidated, what did the validator say?
• Has my submitted task been validated yet?
• How do I request help or a mentor on a task I'm stuck on?
• What projects is my organization currently running?
• Who's the contact for project X?
• How do I register on Tasking Manager?
• Whom should I reach out to for support?
• Where do I file a bug or feature request?
• Where can I give general feedback on the platform?
• How do I become a validator, not just a mapper?
• How do I request project manager rights / create a new project?
• How do I get my organization added to Tasking Manager?
• Where are the tutorials/docs for using iD or JOSM with Tasking Manager?
• Is there a community forum, Slack, or chat I can join?
• How do I report a user?

[ramyaragupathy]

Looping in the FAQ document we built: https://docs.google.com/document/d/1ofcEbVGJOL7CMxcH4FV0toveoZTANGQEyvTbxcs8i2g/edit?tab=t.0

[adiatmad]

I think there is an interesting product/architecture question underneath this proposal: are we building a chatbot that answers Tasking Manager questions, or a conversational interface to Tasking Manager itself?

Looking at the questions already proposed, I think the latter could eventually be much more useful.

The questions seem to fall into different categories:

Knowledge
  → "How do I become a validator?"
  → "Where can I find the JOSM tutorial?"

Current state
  → "What task do I currently have locked?"
  → "Has my submitted task been validated?"
  → "What's project X's current status?"

Historical / user data
  → "How many hours have I contributed?"
  → "What's my mapping streak?"
  → "What projects have I contributed to?"

Derived information
  → "Which projects are short on mappers?"
  → "What projects are trending?"
  → "Which projects match my skill level?"

That suggests that RAG should probably be only one source of truth, rather than the architecture around which everything is built.

A possible conceptual model:

                         User
                           │
                           ▼
                Conversational layer
                           │
             ┌─────────────┼─────────────┐
             ▼             ▼             ▼
         Knowledge      Live TM data   Derived data
            RAG         users/tasks/    analytics/
                        projects/etc.   recommendations
             │             │             │
             └─────────────┼─────────────┘
                           ▼
                    grounded answer

A few things I think are worth defining early

1. Authoritative source

For every representative question, define where the answer should come from.

For example:

• "How do I become a validator?" → documentation
• "What task do I currently have locked?" → current TM state
• "How many hours have I contributed?" → user activity data
• "Why was my task invalidated?" → validation record / validator feedback
• "Which projects are short on mappers?" → current project/task data

This could become part of the evaluation framework rather than evaluating only whether the generated answer sounds good.

2. Identity and authorization

Several proposed questions are user-specific. If the assistant can answer "What task do I currently have locked?" or "What teams am I part of?", it needs to understand the user's identity and inherit the appropriate authorization boundaries from Tasking Manager.

I think this is an important architectural concern to establish early rather than adding permissions after the chatbot is built.

3. Freshness and provenance

There is a big difference between:

"How does task validation work?"

and:

"Is my task currently validated?"

The first can use relatively static documentation. The second requires current state.

It may therefore be useful for the assistant to distinguish static knowledge, current state, historical information, and derived data, and retain enough provenance/freshness information to avoid presenting stale data as current.

4. Answering vs taking actions

Another question I'd clarify early is whether the first version is strictly read-only, or whether the long-term goal includes actions such as opening/locking tasks, joining projects, or requesting help.

I would probably keep the initial version read-only and treat actions as a separate capability with explicit confirmation and stronger authorization requirements.

5. Recommendations

Questions such as "Which projects match my skill level?" or "Recommend a project based on what I usually map" sound like normal chatbot questions, but they actually require explicit criteria.

It may be better to:

TM data
   +
defined criteria
   ↓
candidate projects
   ↓
LLM explains the candidates

rather than allowing the LLM to invent the recommendation logic.

Evaluation could therefore start with a simple matrix

Question
   ↓
Authoritative source
   ↓
Required permissions
   ↓
Freshness requirement
   ↓
Expected answer / refusal behaviour

That would give the project a clearer definition of "grounded" than retrieval accuracy alone.

To me, the interesting long-term opportunity is therefore not simply:

"Ask a chatbot questions about Tasking Manager."

but:

"Use natural language as an interface to Tasking Manager's knowledge, current state, history, and eventually carefully defined actions."

The RAG implementation can then evolve underneath that product model rather than defining the product model itself.

[GLNangle]

I just wanted to weigh in here as I've been working on a JOSM plugin called the HOT Project Companion. There are a lot of features within this, but one which might be interesting (and relevant to this discussion), is a chatbot of sorts that users can ask questions, and it scans the project instructions and finds the answer for them. Obviously, I'm still working on it, but this seemed relevant! :)

I'm sure I could also develop it further to include things outside of the project instructions!

[adiatmad]

One idea I'd like to propose is an AI-assisted review of project instructions before publishing.

Rather than having AI write the instructions from scratch or give them a "quality score", it could act as a second reviewer to flag potential gaps that may cause mapper confusion.

For example:

• Is it clear what should and shouldn't be mapped?
• Are important edge cases or tagging rules missing (e.g., buildings under construction vs. ruins, boundary crossings)?
• Could a first-time mapper misunderstand something?
• Is important information difficult to find or scan?
• Are there recurring questions from similar projects that these instructions could address?

Why Guidance Quality Matters

Recent discussion on the HOT QA/C Working Group highlights how incomplete instructions directly affect the entire workflow:

1. Mapper Frustration & Retention: Incomplete instructions leave mappers guessing on edge cases (e.g., handling buildings under construction, ruins, or power lines). When mappers submit work without clear guidance, their tasks risk being invalidated—which can be extremely demotivating and push volunteers away.
2. Validator Overload: Unclear instructions force validators to fix basic ambiguities or invalidations manually, creating bottlenecks.
3. Usability & Web Content Principles: Mappers scan text online rather than reading long paragraphs. Structuring instructions logically—putting the most critical rules first—ensures essential details aren't missed. Research by the Nielsen Norman Group confirms that concise, scannable web content improves task performance and reduces errors.
4. Structured Creator Prompts: Community members have suggested updating the UI for project creation to include structured "nudges" or prompts (e.g., "Consider: How should ruins or unfinished buildings be handled?"). Providing structured guidance up front makes complete instructions much more likely.

Implementation Approach: AI-Assisted Review vs. Curated Wiki Catalogue

To implement better project guidance, we have two complementary approaches to consider:

Option A: Grounded AI-Assisted QA

Instead of asking a generic AI model whether the text "looks good," the review is grounded in Tasking Manager's accumulated experience, such as:

• Official TM/OSM guidance and project templates.
• Recurring questions, comments, and validator feedback from similar historical projects.
• Patterns from genuinely similar completed projects.

Example output:

"Several mappers asked how to handle buildings crossing the project boundary. Consider adding explicit guidance for this case."

Format: A lightweight MVP provides:

$$\text{Potential gap} \longrightarrow \text{Why it was flagged} \longrightarrow \text{Evidence/source} \longrightarrow \text{Optional suggested wording}$$

We should avoid a numerical score (e.g., "Your instructions are 72% complete"), as instruction quality is contextual and numerical scores create false precision. The project creator remains fully responsible for accepting, editing, or dismissing suggestions.

Option B: Curated Wiki/OSM Project Catalogue

Instead of feeding raw project data directly into an AI model, we create a curated catalogue of high-quality project examples on the HOT TM Wiki or OSM pages.

• Human Curation: Community experts and experienced validators review and curate gold-standard instruction sets.
• Guided Tooling: An AI assistant, form wizard, or creation interface directs project creators to these curated examples based on their specific project type (e.g., disaster response vs. baseline building mapping).
• Tool-Agnostic Setup: Creators can reference proven examples directly without needing complex automated generation.

Long-Term Feedback Loop

Both approaches feed into a valuable feedback loop:

$$\text{Project instructions} \longrightarrow \text{Mapper/Validator feedback} \longrightarrow \text{Recurring problems detected} \longrightarrow \text{Suggested/Wiki guidance updates} \longrightarrow \text{Better reusable templates}$$

Summary

Rather than framing this as "AI-generated project instructions," we should frame it as:

AI-assisted project instruction QA and guided creator tools, grounded in Tasking Manager's accumulated experience and curated community standards.

This creates a smaller, more defensible starting point that reduces mapper friction, improves data quality, and supports project creators without taking away human decision-making.