Skip to content
This repository
branch: master

Dec 20, 2013

  1. Matt Dordal

    Merge pull request #77 from chrisjbray/multi_upstream

    Add a method of specifying multiple upstream hosts in a Network store
    authored December 20, 2013

Dec 17, 2013

  1. Chris Bray

    Add a method of specifying multiple upstream hosts in a Network store

    authored December 10, 2013

Apr 16, 2013

  1. Gautam Roy

    Merge pull request #68 from timjr/master

    the build seems to forget to link some files in shared mode...
    authored April 15, 2013
  2. Gautam Roy

    Merge pull request #66 from mgregson/master

    Build on boost::filesystem v3 and old boost
    authored April 15, 2013
  3. Gautam Roy

    Merge pull request #65 from sproutsocial/master

    Fix to add new line char to meta line, when both options are enabled
    authored April 15, 2013

Feb 07, 2013

  1. the build seems to forget to link some files in shared mode...

    authored February 07, 2013

Jan 25, 2013

  1. Michael Gregson

    Cleaner casing that actually works

    authored January 25, 2013
  2. Michael Gregson

    Port to boost filesystem 3, hopefully keeping support for older boost.

    authored January 25, 2013

Dec 06, 2012

  1. Aaron Rankin

    bug fix: if write_meta is set and add_newlines is enabled, write the …

    …new line char as an EOL for the meta line
    authored December 06, 2012

Jan 06, 2012

  1. Gautam Roy

    Merge pull request #46 from cce/master

    Fix build on newer platforms (latest Mac OS X, Ubuntu 11.10)
    authored January 06, 2012

Oct 28, 2011

  1. chris erway

    fix build on systems with Perl > 5.12.0 (Ubuntu 11.10 Oneiric, Mac OS…

    … X) due to split() incompatibility
    
    thanks mauke from #perl on freenode!
    from http://perldoc.perl.org/perl5140delta.html
    "split() no longer modifies @_ when called in scalar or void context. In void context it now produces a "Useless use of split" warning. This was also a perl 5.12.0 change that missed the perldelta."
    authored October 28, 2011

Jun 27, 2011

  1. Zheng Shao

    Merge pull request #39 from aferreira/patch-1

    Fix minor typo in README
    authored June 26, 2011
  2. Zheng Shao

    Merge pull request #40 from zxvdr/master

    fixed hourly roll period's initial filename
    authored June 26, 2011

May 12, 2011

  1. David Robinson

    fixed hourly roll period's initial filename

    Summary:
    When rotate_period=hourly and no logs exist for a given category,
    the log's filename is created incorrectly. The filename suffix is 0,
    when it should be the current hour.
    authored May 13, 2011

May 04, 2011

  1. Adriano Ferreira

    Fix minor typo in README

    authored May 04, 2011

Oct 15, 2010

  1. Ying-Yi Liang

    Fix compilation issue caused by thrift-0.5.0 changes.

    authored October 14, 2010

Jun 29, 2010

  1. fix filestore::empty() so that bufferstores dont get stuck

    authored June 29, 2010

Jun 08, 2010

  1. Compilation fixes for recent code

    authored June 07, 2010
  2. Small code refactor to help open source push pull

    Summary:
    will remove version number based merge conflicts
    
    Test Plan:
    compile
    
    DiffCamp Revision: 113466
    Reviewed By: pkhemani
    CC: agiardullo, pkhemani, scribe-dev@lists
    Revert Plan:
    OK
    
    git-svn-id: svn+ssh://tubbs/svnapps/fbomb/trunk/fbcode/scribe@27525 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
    authored May 12, 2010 groys committed June 08, 2010

Jun 07, 2010

  1. Merge branch 'opensource_develop_os' into opensource_master

    authored June 07, 2010
  2. honor maxWriteInterval

    Summary:
    because of the way time comparisons were being done maxWriteInterval set to
    1second will actually result in 2s delay.
    
    Test Plan:
    Instrumented the code to check at what times and with what values
    	last_periodic_check and last_handled_messages gets updated. Checked
    	how long the thread stays blocked waiting for work-signal or timeout.
    
    	Verified that there is no thrashing - where the thread keeps
    	continuosuly waking up.
    
    DiffCamp Revision: 120744
    Reviewed By: groys
    CC: agiardullo, pkhemani, groys, scribe-dev@lists
    Tasks:
    #219753: honor maxWriteInterval
    
    Revert Plan:
    OK
    
    git-svn-id: svn+ssh://tubbs/svnapps/fbomb/branches/scribe-os/fbcode/scribe@29223 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
    authored June 04, 2010 groys committed June 07, 2010
  3. Change variable name to comply with fb c++ coding style

    Summary:
    private member variable should be lower camel case with a tailing _
    
    Test Plan:
    compile
    
    DiffCamp Revision: 120003
    Reviewed By: pkhemani
    CC: agiardullo, pkhemani
    Revert Plan:
    OK
    
    git-svn-id: svn+ssh://tubbs/svnapps/fbomb/branches/scribe-os/fbcode/scribe@29008 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
    authored June 02, 2010 groys committed June 07, 2010
  4. Port scribe to facebook ServiceFramework to enable cpprof

    Summary:
    please see task description.
    
    Note: this diff depends on http://tools.intern.facebook.com/D119771
    
    Test Plan:
    http://www.intern.facebook.com/intern/cpprof/profile.php?prof_id=1173&tab=profile_results
    
    DiffCamp Revision: 119553
    Reviewed By: groys
    Commenters: jsong
    CC: agiardullo, zshao, jsong, groys, yliang, scribe-dev@lists
    Tasks:
    
    Revert Plan:
    OK
    
    git-svn-id: svn+ssh://tubbs/svnapps/fbomb/branches/scribe-os/fbcode/scribe@28974 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
    authored June 02, 2010 groys committed June 07, 2010
  5. Separate implementation in env_facebook.h and env_default.h to *.cpp

    Summary:
    The original env_*.h are getting larger and dirtier. We try to make it less
    messy.
    
    Plus, the original env_*.h mixing up the use of namespaces and class scopes.
    This diff
    makes network_config/concurrency/clock... c++ namespaces.
    
    Test Plan:
    1) fbmake dbg then run testsuite
    2) boostrap.sh && make then run testsuite
    
    DiffCamp Revision: 119771
    Reviewed By: groys
    Commenters: jsong
    CC: agiardullo, zshao, jsong, pkhemani, groys, yliang, scribe-dev@lists
    Revert Plan:
    OK
    
    git-svn-id: svn+ssh://tubbs/svnapps/fbomb/branches/scribe-os/fbcode/scribe@28971 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
    authored June 02, 2010 groys committed June 07, 2010
  6. close network store when underlying connection is closed

    Summary:
    as in title. also look at the test plan
    
    Test Plan:
    Earlier:
    	scribe-client running with a buffered store
    	scribe-server running with a file store
    	scribe-client's primary network store points to scribe-server
    
    	step 1 with both scribe client and server running log a message. this
    	populats the scribe-client's connpool
    
    	step 2 bring down scribe server
    
    	step 3 send another message in same category. scribe client goes into
    	disconnected state. periodically it wakes up, goes to SENDING_BUFFER
    	state and then back to DISCONNECTED state. This is because the primary
    	store was never closed. this leads to unnecessary  reading of the
    	backup file
    	[Thu May 27 11:20:41 2010] "[xyz] Changing state from <DISCONNECTED> to
    <SENDING_BUFFER>"
    	[Thu May 27 11:20:41 2010] "[xyz] read <1> entries of <10> bytes from file
    </tmp/corr/xyz/xyz_00000>"
    	Thrift: Thu May 27 11:20:41 2010 TSocket::open() error on socket (after poll)
    <Host: 127.0.0.1 Port: 1464>Connection refused
    	[Thu May 27 11:20:41 2010] "failed to open connection to remote scribe server
    <127.0.0.1:1464> thrift error <socket open() error: Connection refused>"
    	[Thu May 27 11:20:41 2010] "[xyz] choosing new retry interval <1> seconds"
    	[Thu May 27 11:20:41 2010] "[xyz] Changing state from <SENDING_BUFFER> to
    <DISCONNECTED>"
    
    	After the fix:
    	In step 3 the scribe client keeps going from DISCONNECTED to
    	DISCONNECTED
    	Thrift: Thu May 27 11:21:38 2010 TSocket::open() error on socket (after poll)
    <Host: 127.0.0.1 Port: 1464>Connection refused
    	[Thu May 27 11:21:38 2010] "failed to open connection to remote scribe server
    <127.0.0.1:1464> thrift error <socket open() error: Connection refused>"
    	[Thu May 27 11:21:38 2010] "[xyz] choosing new retry interval <1> seconds"
    	[Thu May 27 11:21:38 2010] "[xyz] Changing state from <DISCONNECTED> to
    <DISCONNECTED>"
    
    DiffCamp Revision: 118740
    Reviewed By: groys
    CC: agiardullo, groys, scribe-dev@lists
    Tasks:
    
    Revert Plan:
    OK
    
    Blame Revision:
    26334
    
    git-svn-id: svn+ssh://tubbs/svnapps/fbomb/branches/scribe-os/fbcode/scribe@28771 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
    authored May 28, 2010 groys committed June 07, 2010
  7. make corrupted file handling more robust

    Summary:
    On detecting corruption in a backed up file, stop reading any further.
    Calculate how
    	many bytes are being lost. Output this information in LOG messages and updated
    the
    	bytes lost counter.
    
    	StdFile::readNext can only work for framed files. The dead code for reading
    from non-framed
    	files was anyway wrong - it allocated only 4K for a message line while there
    are many messages
    	longer than that. I removed this dead code.
    
    	I haven't yet added checksums - what the task asks for. I will not close the
    task after this
    	diff
    
    Test Plan:
    1/ Make sure that the regular code when the backup file is not corrupted works.
    
    	2/ Force backup. corrupt backup file by changing frame size field.
    	   move the scribe-server to sending_buffer state. The backup file is removed,
    as much
    	   data that could be sent is sent and log and counters have the loss
    information.
    
    	   [Tue May 25 23:22:35 2010] "WARNING: Corruption Data Loss -14 bytes in
    /tmp/corr/foo/foo_00000"
    	   scribe_overall:bytes lost: 28
    	   scribe_overall:received good: 7
    	   foo:received good: 7
    	   scribe_overall:retries: 154
    	   foo:bytes lost: 28  <===
    	   foo:retries: 154
    	   scribe_overall:sent: 3
    
    	3/ same as 2/ but when uploading the backup file there is an error. The backup
    file is left as
    	   it is. In this situation when the backup file was being read the LOG will
    contain info that
    	   x bytes were lost. But the bytes lost counter won't go up. The bytes lost
    counter only goes
    	   up when the corrupted file is being deleted.
    
    DiffCamp Revision: 118317
    Reviewed By: groys
    CC: agiardullo, pkhemani, groys, scribe-dev@lists
    Tasks:
    
    Revert Plan:
    OK
    
    git-svn-id: svn+ssh://tubbs/svnapps/fbomb/branches/scribe-os/fbcode/scribe@28770 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
    authored May 28, 2010 groys committed June 07, 2010
  8. Extend env_default.h and env_facebook.h to respect implementation dif…

    …ferences
    
    between facebook and the open-source community.
    
    Summary:
    1. put configuration classes in env_*.h in scribe namespace;
    2. add scribe::concurrency for concurrency mechanisms;
    3. make scribe_server.cpp: scribeHandler::scribeHandlerLock configurable;
    4. add scribe::clock for time utilities;
    
    Test Plan:
    just compile it
    
    DiffCamp Revision: 118366
    Reviewed By: groys
    Commenters: jsong
    CC: agiardullo, zshao, jsong, groys, yliang, datafreeway-dev@lists
    Revert Plan:
    OK
    
    git-svn-id: svn+ssh://tubbs/svnapps/fbomb/branches/scribe-os/fbcode/scribe@28661 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
    authored May 27, 2010 groys committed June 07, 2010
  9. Fix testsuite scribe_stop

    Summary:
    Improve error checking in scribeutil.
    
    Test Plan:
    run testsuite
    
    DiffCamp Revision: 110480
    Reviewed By: pkhemani
    Commenters: arivin, jsong
    CC: python-diffs@lists, agiardullo, arivin, jsong, pkhemani, groys,
    scribe-dev@lists
    Revert Plan:
    OK
    
    git-svn-id: svn+ssh://tubbs/svnapps/fbomb/branches/scribe-os/fbcode/scribe@28279 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
    authored May 21, 2010 groys committed June 07, 2010
  10. Revert "Eliminate creation of unnecessary empty files"

    This reverts commit 997408f.
    While, this was aimed at reducing empty files created by closed stores,
    some systems depend on the file being rotated even if a store is closed.
    
    git-svn-id: svn+ssh://tubbs/svnapps/fbomb/branches/scribe-os/fbcode/scribe@28274 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
    authored May 21, 2010 groys committed June 07, 2010
  11. Scribe dynamic bucket id to network store configuration

    Summary:
    this diff added feature such that scribe will be able to work against a
    zookeeper managed network cluster.  Instead of embedding zookeeper into scribe,
    we added a bucketupdater thrift interface that all zookeeper managed hosts
    should support that return a bid to host:port mapping.
    
    added bucketupdate test to testsuites.php.
    
    tested with testsuite.
    
    Test Plan:
    bucketupdater test case added to testsuites.php.
    "
    // testing bucket store updater.
    // Setup:
    // 1. two scribe servers.  one running on port 1465 and using conf file
    //    scribe.conf.bucketupdater.server1, and the other running on port 1466
    //    and using conf file scribe.conf.bucketupdater.server2.
    //    scribe server that configured by scribe.bucketupdater.server1
    //    writes scribe messages to /tmp/scribetest_/bucketupdater/server1
    //    and scribe server that configured by scribe.bucketupdater.server2
    //    writes scribe messages to /tmp/scribetest_/bucketupdater/server2
    // 2. bidupdater server which implements bucket updater interface.
    //    It reads mappings from a local file bidmap which is symbolic
    //    linked to bidmap.1 or bidmap.2.  bidmap.1 maps bucket 1 to
    //    scribe server that writes to /tmp/scribetest_/bucketupdater/server1
    //    and bucket 2 to scribe server that writes to
    /tmp/scribetest_/bucketupdater/server2.
    //    bidmap.2 does the opposite.
    //
    // Test setup:
    // 1. launch scribe -p 1465 scribe.conf.bucketupdater.srever1
    // 2. launch scribe -p 1466 scribe.conf.bucketupdater.server2
    // 3. ln -sf bidmap.1 bidmap.
    // 4. launch bidupdater: bidupdater bidmap
    // 5. scribe -p 1463 scribe.conf.bucketupdater which use bidupdater
    //    to dynamically configure bucket store: bucketupdater.
    // 6. send two message, one with bucket id 1 and the other with bucket id 2
    //    to scribe running on 1463.  Check that the messages are
    //    in /tmp/scribetest_/bucketupdater/server1/bucketupdater/bucket1
    //    and /tmp/scribetest_/bucketupdater/server2/bucketupdater/bucket2
    //    respectively.
    // 7. ln -sf bidmap.2 bid
    // 8. wait for 15 seconds
    // 9. send another two messages, one with bucket 1 and the other bucket 2
    //    Check that messages are in the reverse of 6.
    "
    
    DiffCamp Revision: 108360
    Reviewed By: pkhemani
    Commenters: groys, zshao
    CC: agiardullo, jsong, pkhemani, groys, scribe-dev@lists
    Revert Plan:
    OK
    
    git-svn-id: svn+ssh://tubbs/svnapps/fbomb/branches/scribe-os/fbcode/scribe@28165 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
    authored May 20, 2010 groys committed June 07, 2010
  12. Eliminate creation of unnecessary empty files

    Summary:
    Two changes to help eliminate empty files when running with replay_buffer=no,
    i.e. when we care
    about what data shows up in both primary and secondary
    
    1. eliminate unnecessary periodicCheck calls.
    there are a lot of empty files created due periodicchecks on closed stores
    
    2. We dont take any action on failure to open secondary so trying to do
    that in bufferstore::open is useless
    
    This should also fix the bucketupdater path problem for running
    testsuite
    
    Test Plan:
    testsuite
    
    DiffCamp Revision: 115118
    Reviewed By: jsong
    Commenters: agiardullo
    CC: agiardullo, jsong, groys, scribe-dev@lists
    Revert Plan:
    OK
    
    git-svn-id: svn+ssh://tubbs/svnapps/fbomb/branches/scribe-os/fbcode/scribe@28040 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
    authored May 19, 2010 groys committed June 07, 2010
  13. add scribehtest config to scribe/test

    Summary:
    makes it easy to modify and test locally on dev box clusters
    
    Test Plan:
    none
    
    DiffCamp Revision: 115534
    Reviewed By: srash
    CC: srash, scribe-dev@lists
    Revert Plan:
    OK
    
    git-svn-id: svn+ssh://tubbs/svnapps/fbomb/branches/scribe-os/fbcode/scribe@28024 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
    authored May 19, 2010 groys committed June 07, 2010
  14. Fix compilation bug

    Summary:
    for some reason compiler doesnt understand goto
    
    Test Plan:
    compile
    
    DiffCamp Revision: 114004
    Reviewed By: zshao
    CC: agiardullo, zshao
    Revert Plan:
    OK
    
    git-svn-id: svn+ssh://tubbs/svnapps/fbomb/branches/scribe-os/fbcode/scribe@27670 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
    authored May 13, 2010 groys committed June 07, 2010
  15. Dont accept LOG Calls if scribe server stopping

    Summary:
    were seeing this kind of message on stopping,
    [Mon May 10 17:19:12 2010] "log entry has invalid category
    <nectar_search_typeahead>"
    [Mon May 10 17:19:12 2010] "scribe server exiting"
    [Mon May 10 17:19:12 2010] "log entry has invalid category
    <nectar_search_typeahead>"
    This causes data loss
    
    Test Plan:
    testsuite. scribe_ctrl stop in middle of testsuite, should
    not see these messages
    
    DiffCamp Revision: 113780
    Reviewed By: zshao
    Commenters: pkhemani, agiardullo
    CC: agiardullo, zshao, pkhemani, scribe-dev@lists
    Tasks:
    #206533: Investigate "log entry has invalid category" message
    
    Revert Plan:
    OK
    
    git-svn-id: svn+ssh://tubbs/svnapps/fbomb/branches/scribe-os/fbcode/scribe@27653 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
    authored May 13, 2010 groys committed June 07, 2010
  16. Throw exception if hdfsListDirectory fails

    Summary:
    addendum to Handle failure of listImpl in HDfsFile
    
    Test Plan:
    Test out on the test cluster
    
    DiffCamp Revision: 113697
    Reviewed By: srash
    CC: agiardullo, srash, scribe-dev@lists
    Revert Plan:
    OK
    
    git-svn-id: svn+ssh://tubbs/svnapps/fbomb/branches/scribe-os/fbcode/scribe@27592 2248de34-8caa-4a3c-bc55-5e52d9d7b73a
    authored May 12, 2010 groys committed June 07, 2010
Something went wrong with that request. Please try again.