Contribution workflow: how should contributors coordinate on issues?

[wombatu-kun]

Hi all - I'd like to propose we document a lightweight contribution workflow, mainly around how contributors pick up and coordinate on issues.

What prompted this. A couple of issues ended up with two independent implementations each, which means duplicated effort for contributors and extra triage for maintainers:

• Vector search SQL extension (vector search SQL extension #65): feat: support vector indexes via ALTER TABLE ... CREATE INDEX #449 / feat: add lance_vector_search SQL table-valued function #450 / feat: surface _distance virtual column from vector search #451 and feat: add SQL vector_search table-valued function #574
• Full-text search SQL extension (Full text search SQL extension #234): feat: support full-text search queries via lance_match() SQL function #453 / feat: count(*) through the FTS inverted index #454 / feat: surface the _score virtual column on FTS queries #455 and feat: full text search sql extension #501

Full disclosure: I authored the first set in each pair, so I have a stake here. I'm raising this as a process question, not to argue review priority - I genuinely want to know the preferred way to coordinate so this stops happening.

CONTRIBUTING.md currently covers build, style, docs and release, but doesn't describe how to pick up work, so there's no shared convention today.

Questions for maintainers

1. Is there a preferred signal for "I'm working on this" - self-assigning the issue, or commenting on it?
2. When two PRs target the same issue, how do you decide which to carry forward, and is there anything contributors can do to make that easier?
3. Would you be open to a short "Contribution workflow" section in CONTRIBUTING.md?

A lightweight proposal (just a starting point)

• Before starting non-trivial work, comment on the issue (or self-assign if you have triage rights) so others can see it's taken.
• Link every PR to its issue (Part of #NNN / Closes #NNN) so it shows up under the issue.
• When a duplicate PR appears, maintainers point both authors to the existing issue/PR so they can consolidate.

Happy to open a PR against CONTRIBUTING.md if there's appetite. Since CONTRIBUTING.md is auto-published to the Lance contributing guide, getting this right helps the whole project, not just this repo.

[hamersaw]

@wombatu-kun really appreciate opening this discussion and for your continued stewardship of the Lance Spark connector. I think this repo has grown beyond the ad-hoc development style that has worked until recently and some stronger guides on community and contributions will help keep everything organized. I'll start out with my takes on your questions / proposal and then provide a few of my own.

Addressing Questions

Is there a preferred signal for "I'm working on this" - self-assigning the issue, or commenting on it?

I think your proposal of having an issue and having PRs linked to that and / or having the issue assigned to a user is the canonical way to solve this problem. We can still end up with duplicate implementations, but the singular issue should be the root construct to rectify differences.

When two PRs target the same issue, how do you decide which to carry forward, and is there anything contributors can do to make that easier?

In my thoughts down below I outlined a "design proposal" process. For something to have a duplicate implementation I think it needs to be big enough that there even can be significant implementation differences. A design proposal should be reasonable for multiple parties to weight in on what the end solution looks like and incremental delivery to get there. It has the added benefit of providing early visibility and feedback for the developer.

Would you be open to a short "Contribution workflow" section in CONTRIBUTING.md?

IMO this is exactly what we have to do moving forward. Specifically, be explicit about what we need to do to handle scaling contributions as the project progresses.

My additional thoughts:

• Design proposals for large features: The biggest problem we're facing as maintainers, and that is certainly echoed across the OSS space, is that AI produces LoC faster than is reasonable to review. Often times this solution has some base assumptions that may skew from what we might have chosen, fixing those requires a complete rearchitect and iterations of this make the PR difficult to reasonably iterate on. IMO we need to introduce some sort of a design proposal process for large features - this way we can iron out the implementation approach before sifting through code.

• Maintainers responsibilities: As maintainers it should be our responsibility to groom the issues / PRs and ensure that duplication does not become rampant. This is something we need to be better on to ensure the projects continual success. I think specific contributing guidelines (ex. PRs linked to issues, design proposals for large efforts, etc) is the best way for the community to help with this.

[hamersaw]

cc @jackye1995 @geruh @wjones127 @westonpace

[geruh]

+1 from me!

I think visibility is the biggest thing here.

There's still a lot of work to do to get the Spark connector closer to not only feature parity, but surfacing all lance capabilities in the spark engine. Which means there are going to be plenty of opportunities for contributors to jump in. The more visible we can be about what we're working on, the less likely we are to duplicate effort or miss something important.

Things like commenting on issues, linking PRs back to issues, design discussions for larger features, the community sync, and the Dev list all help get more eyes on the work and keep everyone moving in the same direction.

To add onto what Dan said, I think here it's not about avoiding duplicate work. But it's making sure all the contributors and maintainers have enough visibility into whats going on across the project.

[hamersaw]

cc @ivscheianu @LuciferYang @fangbo @summaryzb as well - very interested in your thoughts!

[LuciferYang]

Thanks for starting this @wombatu-kun. +1 to the design-proposal idea from @hamersaw, and I think @geruh is right that it really comes down to visibility. We've dealt with all of this on Spark over the years, so let me share what's actually worked there and what hasn't — some of it is worth copying, some of it we should probably do a lighter version of given this repo is smaller and lives on GitHub.

On linking PRs to issues: Spark requires a JIRA for every change and the PR title has to be [SPARK-XXXXX][COMPONENT] …, with a bot that rejects anything missing it. The JIRA part doesn't matter for us, but the enforcement part does. If we just ask people to link their PRs, it'll happen for a while and then quietly stop. I'd make Closes #NNN / Part of #NNN in the PR body actually required via a small CI check, with trivial fixes exempted.

On design proposals for bigger features: this is the SPIP process in Spark — a short doc that answers a few fixed questions (what are you solving, how's it done today, what's new, who does it affect, what are the risks, how would you ship an MVP), plus a maintainer who sponsors it and a discussion before any code gets written. Honestly this is the main thing that helps with the "AI writes more code than anyone can review" problem, because you sort out the approach before the diff exists. Two things I'd carry over from Spark's experience:

• Pick a sponsor/reviewer up front. When the contributor knows who's going to review and merge it, and that person has already agreed on the shape of the thing, you avoid most of those painful "great, now rewrite the whole thing" rounds.
• Put a clock on it. The way SPIPs go wrong is they just sit there with nobody responding, and the contributor is stuck. Even a rough "expect maintainer feedback within a week" helps.

I'd keep this only for genuinely large stuff though — a new SQL extension, a new public API, anything cross-cutting. Normal issues should stay normal issue→PR with no ceremony. A markdown file under docs/proposals/ or just a design label is enough; no need for a formal vote at this point.

On claiming issues: comment on the issue before you start something non-trivial, or assign it to yourself if you can. The catch — and we see this constantly on Spark — is people claim things and then disappear, and now the issue looks taken but nothing's happening. So we should also say out loud that a claim goes stale: if there's no PR or activity after a couple of weeks, anyone can pick it up. Worth stating explicitly so nobody feels like they're cutting in line.

On big PRs: Spark splits large work into an umbrella JIRA with sub-tasks, each a small PR. Same idea here — umbrella issue with a checklist, each PR marked Part of #NNN. A 300-line PR I can actually review beats a 3,000-line one I have to push back on wholesale, and you find out early if the direction's off.

And when two PRs do show up for the same thing, we'll need a couple of practical rules so the call doesn't feel arbitrary. By default, if the earlier-submitted PR doesn't have any critical design flaws, we review and merge that one first — first in, first served, basically. The one exception: when it's a maintainer's PR up against a new contributor's PR for the same issue, we'll prioritize reviewing the newcomer's, as long as there's no fundamental design problem with it. We'd rather use these as chances to pull external contributors in than have maintainers absorb all the work. Either way, whoever makes the call should write the reasoning on the issue — as the person on one side of these duplicate pairs, the part that actually stings isn't losing out, it's not knowing why.

[ivscheianu]

Great ideas and discussion, thanks a lot for bringing this up. I really think that formalizing this process a bit will help us streamline the development flow and move faster in the long run, especially as the project grows and more contributors get involved.

I really resonate with the ideas from @hamersaw and @LuciferYang. The design-doc approach with an assigned sponsor/maintainer makes a lot of sense for big features. It gives everyone visibility into what's being worked on, surfaces design disagreements early, and creates a good coordination point. Combined with the claim-expiry idea and PR-to-issue linking, I think that strikes the right balance between structure and staying lightweight. The point about putting a clock on maintainer feedback is also valid. Contributors investing significant time deserve a timely signal on whether the direction is right. Even a rough SLA like "initial feedback within a week" goes a long way in keeping momentum.

One question: where should the initial design proposal live? I'm thinking the simplest option is just writing it directly in the issue itself. That keeps the discussion, the design, and the linked PRs all in one place. What do you think?

Happy to adhere to the new process!

[fangbo]

I'm thinking the simplest option is just writing it directly in the issue itself. That keeps the discussion, the design, and the linked PRs all in one place.

+1

[wombatu-kun]

Thanks everyone for the thoughtful discussion - @hamersaw, @geruh, @LuciferYang, @ivscheianu, @fangbo.

I've summarized the consensus into a new "Contribution Workflow" section in CONTRIBUTING.md and opened a PR: #627

It covers: claiming issues (comment/self-assign, claims go stale after ~2 weeks), linking every PR to its issue (Closes/Part of), design proposals for large features written in the issue with a maintainer sponsor and a ~1-week feedback target, keeping PRs small under an umbrella issue, the default first-in-first-served rule for duplicate PRs (with priority for new contributors over maintainers) and recording the reasoning on the issue, and maintainer grooming responsibilities.

Please review and suggest edits on the PR. Once it lands, let's treat this as our shared workflow and keep coordinating in the open. Thanks again for working through this together.