Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

zoekt: support indexing multiple branches in zoekt-archive-index #7930

Closed
keegancsmith opened this issue Jan 21, 2020 · 8 comments
Closed

zoekt: support indexing multiple branches in zoekt-archive-index #7930

keegancsmith opened this issue Jan 21, 2020 · 8 comments
Assignees
@keegancsmith
Labels
customer Important issues reported or desired by a customer. estimate/2d feature-request planned/3.13 Issues that were planned for the given milestone. Used by cmd/tracking-issue. spike Time boxed investigation meant to facilitate more granular planning.
Milestone
Backlog

Comments

@keegancsmith
Copy link
Member

keegancsmith commented Jan 21, 2020 •
edited

Parent issue: #6728

The first step in supporting indexing multiple branches is for our indexing method to handle multiple branches. Right now we index tarballs via zoekt-archive-index. This issue is about supporting passing multiple tarballs (or zips) into zoekt-archive-index such that each tarball represents a branch. See the parent issue for much more context/discussion on this topic.

Requested by https://app.hubspot.com/contacts/2762526/company/554338610
@keegancsmith keegancsmith added feature-request customer Important issues reported or desired by a customer. team/core-services estimate/2d labels Jan 21, 2020
@keegancsmith keegancsmith added this to the 3.13 milestone Jan 21, 2020
@keegancsmith keegancsmith self-assigned this Jan 21, 2020
@keegancsmith keegancsmith mentioned this issue Jan 21, 2020
Closed
@tsenart
Copy link
Contributor

tsenart commented Jan 22, 2020

A potential idea for consideration is to have git-server merge these archives itself and serve that result for download. Not sure if worth it from a performance / implementation time perspective, just wanted to leave this here :-)

http://tldp.org/HOWTO/SquashFS-HOWTO/whatis.html

@keegancsmith
Copy link
Member Author

I do like that idea a lot, and was in fact something I mentioned in my original proposal. I didn’t move forward with the idea due to it being more effort. But I am quite enamoured with the idea of having more control of this bit of data we generate and use everywhere.

@keegancsmith
Copy link
Member Author

@tsenart filed this issue to explore your idea. Something to consider for next milestone #7937

@keegancsmith keegancsmith mentioned this issue Jan 22, 2020
Closed
@keegancsmith
Copy link
Member Author

Most of my work I have been doing by submitting patches upstream. However, I have now created a draft PR with active work. I have spent the better part of two days hacking on this, which is more time than we allocated for this spike. We have something working, but very hacky. I would regard this spike as complete, but would like to clean up the code a bit since it is extremely hacky at the moment. So will leave this issue open and address this later in the cycle.

sourcegraph/zoekt#36

@tsenart tsenart mentioned this issue Jan 23, 2020
Closed
81 tasks
@tsenart tsenart added the planned/3.13 Issues that were planned for the given milestone. Used by cmd/tracking-issue. label Jan 24, 2020
@keegancsmith keegancsmith added the spike Time boxed investigation meant to facilitate more granular planning. label Jan 27, 2020
@uwedeportivo
Copy link
Contributor

Dear all,

This is your release captain speaking. 🚂🚂🚂

Branch cut for the 3.13 release is scheduled for tomorrow.

Is this issue / PR going to make it in time? Please change the milestone accordingly.
When in doubt, reach out!

Thank you

@tsenart tsenart modified the milestones: 3.13, Backlog Feb 20, 2020
slimsag added a commit that referenced this issue Apr 10, 2020
@slimsag
observability: convert Symbols dashboard into generated form
eb3b75b 
Before:

![image](https://user-images.githubusercontent.com/3173176/78980368-9d5c1580-7ad2-11ea-9d82-8a90f322505e.png)

After:

![image](https://user-images.githubusercontent.com/3173176/78980386-a5b45080-7ad2-11ea-8956-e2066faddaac.png)

Fixes #7930
Fixes #9731
Fixes #9732
@slimsag slimsag mentioned this issue Apr 10, 2020
Merged
@tsenart
Copy link
Contributor

tsenart commented Apr 10, 2020

@slimsag: I believe this was closed by mistake?

@tsenart tsenart reopened this Apr 10, 2020
@slimsag
Copy link
Member

slimsag commented Apr 10, 2020

Yes, mistake, thanks! I corrected the typo in my PR description but not in the commit and didn't realize it.

@keegancsmith keegancsmith mentioned this issue Apr 15, 2020
Closed
@keegancsmith
Copy link
Member Author

Using shallow git clones instead. See #9888 and #6728

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@keegancsmith keegancsmith

Labels
customer Important issues reported or desired by a customer. estimate/2d feature-request planned/3.13 Issues that were planned for the given milestone. Used by cmd/tracking-issue. spike Time boxed investigation meant to facilitate more granular planning.
Projects
None yet
Milestone
Backlog
Development

Successfully merging a pull request may close this issue.

zoekt-archive-index: Support multiple archives / branches. sourcegraph/zoekt
4 participants
@tsenart @keegancsmith @uwedeportivo @slimsag