[SUPPORT] [Question]Code Logic in RocksDbBasedFileSystemView

[zhangyue19921010]

Here is Override function fetchLatestBaseFile(String partitionPath, String fileId) in RocksDbBasedFileSystemView

  @Override
  protected Option<HoodieBaseFile> fetchLatestBaseFile(String partitionPath, String fileId) {
    // Retries only file-slices of the file and filters for the latest
    return Option
        .ofNullable(rocksDB
            .<HoodieBaseFile>prefixSearch(schemaHelper.getColFamilyForView(),
                schemaHelper.getPrefixForDataFileViewByPartitionFile(partitionPath, fileId))
            .map(Pair::getValue).reduce(null,
                (x, y) -> ((x == null) ? y
                    : (y == null) ? null
                        : HoodieTimeline.compareTimestamps(x.getCommitTime(), HoodieTimeline.GREATER_THAN, y.getCommitTime())
                            ? x
                            : y)));
  }

Question is Why we need to OverWrite this un-abstract function in AbstractTableFileSystemView

  /**
   * Default implementation for fetching latest base-file.
   *
   * @param partitionPath Partition path
   * @param fileId File Id
   * @return base File if present
   */
  protected Option<HoodieBaseFile> fetchLatestBaseFile(String partitionPath, String fileId) {
    return Option.fromJavaOptional(fetchLatestBaseFiles(partitionPath)
        .filter(fs -> fs.getFileId().equals(fileId)).findFirst());
  }

Is it necessary? Or Is it correctly if we don't flow this common logic.

[zhangyue19921010]

@bvaradar Sorry to bother you. Would you mind to take a look？
Really appreciate it!

[nsivabalan]

this is my understanding by looking at the code of interest. will wait to hear from Balaji though.
Its more of an optimization. If we don't override, we call into fetchLatestBaseFiles(partitionPath) which in turn calls into fetchAllStoredFileGroups(partitionPath). so this is essentially fetching all file groups for the partition of interest and then deducing from it.

where as in our overridden method, we filter directly w/ rocksDB for the partition and fileId of interest and then fetching just the latest from it, which I feel might be faster.

[nsivabalan]

@zhangyue19921010 : do you need more clarifications ? or you wanna wait to hear from Balaji.

[zhangyue19921010]

@nsivabalan Thanks for your attention. We can close this Support :)