Subdir templates #34

Closed
wants to merge 2 commits into
from

Conversation

Projects
None yet
3 participants

Hi,

I made some tweaks to allow for subdirectories to be defined by using ${YEAR}, ${MONTH}, ${DAY}, and ${HOUR} which allows for simple partitioning with HDFS and whatnot. It is much more flexible than the "store_tree" change that was created a while ago (and then looks like it was removed).

If you'd like me to make some changes just let me know.

Thanks,
Mike

Mike Lewis added some commits Nov 9, 2010

Mike Lewis Added boost regex library to support filename templates 0acb954
Mike Lewis Modified FileStoreBase to support using subdirectories based on curre…
…nt date

This involved the following changes:
  * remove filePath from FileStoreBase
    and replaced with method makeFilePath
    which takes a tm object
  * make every function that formerly
    referenced filePath to reference require
    a tm argument

This will subsitute ${YEAR}, ${MONTH}, ${DAY}, and ${HOUR} in
the 'sub_directory' argument in the configuration file. ex:

  sub_directory=${YEAR}/${MONTH}/${DAY}

Or if for having it work with hive partitions:

  sub_directory=dt=${YEAR}${MONTH}${DAY}
5f87c08

Hey this is a test

Hi Mike,
this is a very interesting feature and we're considering adding the patch to our scribe log server instance, but I have a comment :)

AFAIUI once the patch is in, all the paths for a hourly rotated store are split in subdirs, while I see cases where one want to choose to enable this method only in some stores (the ones with more traffic/data) while leaving the default behavior in the others.

So, would you consider adding a config option to enable/disable the "split paths" in the store configuration? I don't have the coding skills to do that myself, hence why I'm asking :)

Contributor

jamesgpearce commented May 29, 2014

Facebook has not maintained or supported Scribe externally for some time, and we are closing its old and outstanding pull requests.

Many, many thanks for your support of the project. If you have any further questions, please don't hesitate to let me know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment