Skip to content

MB-62182: Implement File transfer APIs#2279

Merged
Thejas-bhat merged 5 commits intomasterfrom
fileTransfer
Mar 27, 2026
Merged

MB-62182: Implement File transfer APIs#2279
Thejas-bhat merged 5 commits intomasterfrom
fileTransfer

Conversation

@Thejas-bhat
Copy link
Copy Markdown
Member

@Thejas-bhat Thejas-bhat commented Jan 29, 2026

  • Utilizes the new file transfer APIs to able to transfer the trained index file from one bleve index to another. This is helpful in a multi-partition scenario with a single datasource - where only one bleve index is trained on a set of vectors and then transfer the generated file to the other bleve indexes.
  • The file transfer is currently focused around the trained index file but the protocol is such that its future extendable for other files if needed.
  • The main control flow here is that the application needs to implement the IndexDirectory interface which maintains a reference to the destination bleve index.
    • Now, when the srcIndex.CopyFile(index.TrainedIndexFileName, directory) is invoked, it internally fetches the writer on the destination using directory.GetWriter()
    • Then, as part of the CopyFile it does an io.Copy() of the specific file and currently this is focused only for trained index file.
    • This is followed by an update bolt transaction + commit and sync in the destination index by invoking directory.SetPathInBolt(fileName, filePath). This is mandatory since bolt is reflective of the on-disk information.

@Thejas-bhat Thejas-bhat force-pushed the fileTransfer branch 3 times, most recently from 925464a to 54194a5 Compare January 29, 2026 21:59
@Thejas-bhat Thejas-bhat moved this from Todo to In Progress in Fast Merge Jan 30, 2026
@Thejas-bhat Thejas-bhat force-pushed the fastmerge branch 2 times, most recently from 74072d3 to ce43814 Compare February 6, 2026 00:24
@Thejas-bhat Thejas-bhat changed the base branch from fastmerge to fmflag February 17, 2026 18:16
@Thejas-bhat Thejas-bhat force-pushed the fileTransfer branch 2 times, most recently from be7a3f5 to 87daec7 Compare February 19, 2026 18:09
@Thejas-bhat Thejas-bhat changed the base branch from fmflag to fastmerge February 24, 2026 19:24
@Thejas-bhat Thejas-bhat force-pushed the fastmerge branch 4 times, most recently from 8ad6a31 to b0c71ba Compare March 23, 2026 23:05
@Thejas-bhat Thejas-bhat force-pushed the fileTransfer branch 2 times, most recently from f3545e0 to 443ce94 Compare March 25, 2026 18:13
@abhinavdangeti abhinavdangeti added this to the v2.6.0 milestone Mar 25, 2026
@Thejas-bhat Thejas-bhat force-pushed the fastmerge branch 2 times, most recently from d526196 to 02394a9 Compare March 26, 2026 00:15
@Thejas-bhat Thejas-bhat force-pushed the fastmerge branch 2 times, most recently from f4bae39 to 53a1ad0 Compare March 26, 2026 22:25
Base automatically changed from fastmerge to master March 27, 2026 18:47
@abhinavdangeti
Copy link
Copy Markdown
Member

@Thejas-bhat this requires rebase and conflict resolution, also address any outstanding comments would you.

@coveralls
Copy link
Copy Markdown

coveralls commented Mar 27, 2026

Coverage Status

coverage: 52.497% (-0.1%) from 52.61%
when pulling 7ac845f on fileTransfer
into 3d4b002 on master.

abhinavdangeti
abhinavdangeti previously approved these changes Mar 27, 2026
@abhinavdangeti abhinavdangeti self-requested a review March 27, 2026 20:35
@Thejas-bhat Thejas-bhat merged commit 007a5da into master Mar 27, 2026
10 checks passed
@Thejas-bhat Thejas-bhat deleted the fileTransfer branch March 27, 2026 20:49
@github-project-automation github-project-automation bot moved this from In Progress to Done in Fast Merge Mar 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants