Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Some cleanup now that it's the external repo #2

Closed
wants to merge 1 commit into from
Closed

Some cleanup now that it's the external repo #2

wants to merge 1 commit into from

Conversation

chipturner
Copy link
Contributor

Needed a .gitignore and folly-config.h is generated by configure so isn't needed in this repo.

@jdelong jdelong closed this Jun 5, 2012
@jdelong
Copy link
Contributor

jdelong commented Jun 5, 2012

Merged upstream and pushed

@jdelong jdelong mentioned this pull request Mar 23, 2014
Closed
ghost pushed a commit that referenced this pull request Sep 15, 2015
@luciang
folly: async: fix SIOF in test
062f90e 
Summary: avoid sillyness:

```
$ _build/opt/folly/io/async/test/async_test --gtest_list_tests
ASAN:SIGSEGV
=================================================================
==3245135==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000010 (pc 0x000000583444 sp 0x7fff17ba0c40 bp 0x7fff17ba0c80 T0)
    #0 0x583443 in std::_Rb_tree<int, std::pair<int const, folly::SSLContext::SSLLockType>, std::_Select1st<std::pair<int const, folly::SSLContext::SSLLockType> >, std::less<int>, std::allocator<std::pair<int const, folly::SSLContext::SSLLockType> > >::operator=(std::_Rb_tree<int, std::pair<int const, folly::SSLContext::SSLLockType>, std::_Select1st<std::pair<int const, folly::SSLContext::SSLLockType> >, std::less<int>, std::allocator<std::pair<int const, folly::SSLContext::SSLLockType> > > const&) third-party2/libgcc/4.9.x/gcc-4.9-glibc-2.20-fb/024dbc3/include/c++/4.9.x/bits/stl_tree.h:1138
    #1 0x583443 in std::map<int, folly::SSLContext::SSLLockType, std::less<int>, std::allocator<std::pair<int const, folly::SSLContext::SSLLockType> > >::operator=(std::map<int, folly::SSLContext::SSLLockType, std::less<int>, std::allocator<std::pair<int const, folly::SSLContext::SSLLockType> > > const&) third-party2/libgcc/4.9.x/gcc-4.9-glibc-2.20-fb/024dbc3/include/c++/4.9.x/bits/stl_map.h:295
    #2 0x583443 in folly::SSLContext::setSSLLockTypes(std::map<int, folly::SSLContext::SSLLockType, std::less<int>, std::allocator<std::pair<int const, folly::SSLContext::SSLLockType> > >) folly/io/async/SSLContext.cpp:682
    #3 0x40e028 in Initializer folly/io/async/test/AsyncSSLSocketTest2.cpp:143
    #4 0x40e028 in __static_initialization_and_destruction_0 folly/io/async/test/AsyncSSLSocketTest2.cpp:146
    #5 0x40e028 in _GLOBAL__sub_I__ZN5folly47AsyncSSLSocketTest2_AttachDetachSSLContext_Test10test_info_E folly/io/async/test/AsyncSSLSocketTest2.cpp:147
    #6 0x66cf2e in __libc_csu_init /home/engshare/third-party2/glibc/2.20/src/glibc-2.20/csu/elf-init.c:88
    #7 0x7f7145600084 in __libc_start_main (/usr/local/fbcode/gcc-4.9-glibc-2.20-fb/lib/libc.so.6+0x20084)
    #8 0x410be5 (/data/users/lucian/fbcode2/_build/opt/folly/io/async/test/async_test+0x410be5)

AddressSanitizer can not provide additional info.
AAAAAAASUMMARY: AddressSanitizer: SEGV third-party2/libgcc/4.9.x/gcc-4.9-glibc-2.20-fb/024dbc3/include/c++/4.9.x/bits/stl_tree.h:1138 std::_Rb_tree<int, std::pair<int const, folly::SSLContext::SSLLockType>, std::_Select1st<std::pair<int const, folly::SSLContext::SSLLockType> >, std::less<int>, std::allocator<std::pair<int const, folly::SSLContext::SSLLockType> > >::operator=(std::_Rb_tree<int, std::pair<int const, folly::SSLContext::SSLLockType>, std::_Select1st<std::pair<int const, folly::SSLContext::SSLLockType> >, std::less<int>, std::allocator<std::pair<int const, folly::SSLContext::SSLLockType> > > const&)
==3245135==ABORTING
```

Reviewed By: @philippv

Differential Revision: D2440796
ghost pushed a commit that referenced this pull request Sep 17, 2015
folly Singleton: "eager" option to initialize upfront
e4527fb 
Summary: Instead of the default lazy-loading behavior (still the default) some singletons might need to get initialized at startup time.  This would be for singletons which take a long time for the instance's constructor to run, e.g. expensive initialization by reading some large dataset, talking to an outside service, and so on.

Provides a way for singletons to opt-in to this, and get populated at the time  `registrationComplete()` is called, instead of lazily.

Some notes about the way I implemented here, mainly, why I did this as a "builder-pattern" kind of thing and not some other way.  I could probably be convinced to do otherwise. :)

* Changing the constructor: the constructor's already slightly fiddly with the two optional -- well, one optional construct function, and another optional-but-only-if-construct-provided, destruct function.  Didn't want to pile more into the ctor.
* New superclass called `EagerLoadedSingleton`; just didn't want to add more classes, esp. if it's just to add one more option.
* Method like `void setEagerLoad()` that makes this eager-load; not sure where one would write the `shouldEagerLoad()` call, probably in some central initialization spot in `main()`, but all the maintenance would have to go there.  I like that it's "attached" to the singleton being defined.  (Though you can still do this.)  Bonus #2; the rule that builds the cpp containing "main" doesn't need to import this dependency and the cpp doesn't have to include Singleton just to do this eager-load call, nor the header for the type itself.
* Omitting this altogether and just saying `folly::Singleton<Foo>::get_weak()` to "ping" the singleton and bring into existence: see last point.  Still might need to have the file containing this initialization decorum include/link against Foo, as well as have one place to maintain the list of things to load up-front.

Reviewed By: @meyering

Differential Revision: D2449081
korovkin pushed a commit to korovkin/folly that referenced this pull request Oct 1, 2015
@luciang @korovkin
folly: async: fix SIOF in test
fdc193b 
Summary: avoid sillyness:

```
$ _build/opt/folly/io/async/test/async_test --gtest_list_tests
ASAN:SIGSEGV
=================================================================
==3245135==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000010 (pc 0x000000583444 sp 0x7fff17ba0c40 bp 0x7fff17ba0c80 T0)
    #0 0x583443 in std::_Rb_tree<int, std::pair<int const, folly::SSLContext::SSLLockType>, std::_Select1st<std::pair<int const, folly::SSLContext::SSLLockType> >, std::less<int>, std::allocator<std::pair<int const, folly::SSLContext::SSLLockType> > >::operator=(std::_Rb_tree<int, std::pair<int const, folly::SSLContext::SSLLockType>, std::_Select1st<std::pair<int const, folly::SSLContext::SSLLockType> >, std::less<int>, std::allocator<std::pair<int const, folly::SSLContext::SSLLockType> > > const&) third-party2/libgcc/4.9.x/gcc-4.9-glibc-2.20-fb/024dbc3/include/c++/4.9.x/bits/stl_tree.h:1138
    facebook#1 0x583443 in std::map<int, folly::SSLContext::SSLLockType, std::less<int>, std::allocator<std::pair<int const, folly::SSLContext::SSLLockType> > >::operator=(std::map<int, folly::SSLContext::SSLLockType, std::less<int>, std::allocator<std::pair<int const, folly::SSLContext::SSLLockType> > > const&) third-party2/libgcc/4.9.x/gcc-4.9-glibc-2.20-fb/024dbc3/include/c++/4.9.x/bits/stl_map.h:295
    facebook#2 0x583443 in folly::SSLContext::setSSLLockTypes(std::map<int, folly::SSLContext::SSLLockType, std::less<int>, std::allocator<std::pair<int const, folly::SSLContext::SSLLockType> > >) folly/io/async/SSLContext.cpp:682
    facebook#3 0x40e028 in Initializer folly/io/async/test/AsyncSSLSocketTest2.cpp:143
    facebook#4 0x40e028 in __static_initialization_and_destruction_0 folly/io/async/test/AsyncSSLSocketTest2.cpp:146
    facebook#5 0x40e028 in _GLOBAL__sub_I__ZN5folly47AsyncSSLSocketTest2_AttachDetachSSLContext_Test10test_info_E folly/io/async/test/AsyncSSLSocketTest2.cpp:147
    facebook#6 0x66cf2e in __libc_csu_init /home/engshare/third-party2/glibc/2.20/src/glibc-2.20/csu/elf-init.c:88
    facebook#7 0x7f7145600084 in __libc_start_main (/usr/local/fbcode/gcc-4.9-glibc-2.20-fb/lib/libc.so.6+0x20084)
    facebook#8 0x410be5 (/data/users/lucian/fbcode2/_build/opt/folly/io/async/test/async_test+0x410be5)

AddressSanitizer can not provide additional info.
AAAAAAASUMMARY: AddressSanitizer: SEGV third-party2/libgcc/4.9.x/gcc-4.9-glibc-2.20-fb/024dbc3/include/c++/4.9.x/bits/stl_tree.h:1138 std::_Rb_tree<int, std::pair<int const, folly::SSLContext::SSLLockType>, std::_Select1st<std::pair<int const, folly::SSLContext::SSLLockType> >, std::less<int>, std::allocator<std::pair<int const, folly::SSLContext::SSLLockType> > >::operator=(std::_Rb_tree<int, std::pair<int const, folly::SSLContext::SSLLockType>, std::_Select1st<std::pair<int const, folly::SSLContext::SSLLockType> >, std::less<int>, std::allocator<std::pair<int const, folly::SSLContext::SSLLockType> > > const&)
==3245135==ABORTING
```

Reviewed By: @philippv

Differential Revision: D2440796
korovkin pushed a commit to korovkin/folly that referenced this pull request Oct 1, 2015
@korovkin
folly Singleton: "eager" option to initialize upfront
a625f22 
Summary: Instead of the default lazy-loading behavior (still the default) some singletons might need to get initialized at startup time.  This would be for singletons which take a long time for the instance's constructor to run, e.g. expensive initialization by reading some large dataset, talking to an outside service, and so on.

Provides a way for singletons to opt-in to this, and get populated at the time  `registrationComplete()` is called, instead of lazily.

Some notes about the way I implemented here, mainly, why I did this as a "builder-pattern" kind of thing and not some other way.  I could probably be convinced to do otherwise. :)

* Changing the constructor: the constructor's already slightly fiddly with the two optional -- well, one optional construct function, and another optional-but-only-if-construct-provided, destruct function.  Didn't want to pile more into the ctor.
* New superclass called `EagerLoadedSingleton`; just didn't want to add more classes, esp. if it's just to add one more option.
* Method like `void setEagerLoad()` that makes this eager-load; not sure where one would write the `shouldEagerLoad()` call, probably in some central initialization spot in `main()`, but all the maintenance would have to go there.  I like that it's "attached" to the singleton being defined.  (Though you can still do this.)  Bonus facebook#2; the rule that builds the cpp containing "main" doesn't need to import this dependency and the cpp doesn't have to include Singleton just to do this eager-load call, nor the header for the type itself.
* Omitting this altogether and just saying `folly::Singleton<Foo>::get_weak()` to "ping" the singleton and bring into existence: see last point.  Still might need to have the file containing this initialization decorum include/link against Foo, as well as have one place to maintain the list of things to load up-front.

Reviewed By: @meyering

Differential Revision: D2449081
ghost pushed a commit that referenced this pull request Apr 28, 2016
@abirchall
Create ReadHolder::unlock
95e7a3b 
Summary:
Currently you need to depend on the destructor of `ReadHolder` (using closures as in code block #1 below or empty assignment as in code block #2 below) to ensure that a `ReadHolder` goes out of scope (and unlocks) in order to subsequently acquire a write lock via `WriteHolder` without deadlocking.
This diff introduces a way of unlocking a `ReadHolder` while it's still in scope such that a `WriteHolder` can be acquired. This makes the code more straight forward (reducing the risk of deadlock due to a programmer's misunderstanding of how `SharedMutex` / the holder framework works) => see code block # 3 below
Also add some documentation about why `WriteHolder::WriteHolder(ReadHolder&&)` doesn't exist

Code Block #1 : Use of closures
```
class foo {
 public:
  std::string getMemoizedData() {
    {
      folly::SharedMutex::ReadHolder readHolder(lock_);
      if (!data_.empty()) {
        // important to return by value, otherwise caller might access
        // data_ after we release the read lock
        return data_;
      }
    }
    {
      // try again with a write lock
      folly::SharedMutex::WriteHolder writeHolder(lock_);
      if (data_.empty()) {
        data_ = "my awesome string";
      }
      return data_;
    }
  }

 private:
  folly::SharedMutex lock_;
  std::string data_;
};
```

Code Block #2 : Use of empty assignment
```
class foo {
 public:
  std::string getMemoizedData() {
    folly::SharedMutex::ReadHolder readHolder(lock_);
    if (!data_.empty()) {
      // important to return by value, otherwise caller might access
      // data_ after we release the read lock
      return data_;
    }
    readHolder = {};

    // try again with a write lock
    folly::SharedMutex::WriteHolder writeHolder(lock_);
    if (data_.empty()) {
      data_ = "my awesome string";
    }
    return data_;
  }

 private:
  folly::SharedMutex lock_;
  std::string data_;
};
```

Code Block #3 : Use of unlock()
```
class foo {
 public:
  std::string getMemoizedData() {
    folly::SharedMutex::ReadHolder readHolder(lock_);
    if (!data_.empty()) {
      // important to return by value, otherwise caller might access
      // data_ after we release the read lock
      return data_;
    }
    readHolder->unlock();

    // try again with a write lock
    folly::SharedMutex::WriteHolder writeHolder(lock_);
    if (data_.empty()) {
      data_ = "my awesome string";
    }
    return data_;
  }

 private:
  folly::SharedMutex lock_;
  std::string data_;
};
```

Reviewed By: yfeldblum

Differential Revision: D3176025

fb-gh-sync-id: c7d47ca71df08673c7c1f1fd5ed9e01a663c1797
fbshipit-source-id: c7d47ca71df08673c7c1f1fd5ed9e01a663c1797
ghost pushed a commit that referenced this pull request May 19, 2016
@luciang
folly: FBString: avoid ubsan in assert
4d78cf4 
Summary:
```
buck-out/gen/folly/__default_headers__#default,headers/folly/FBString.h:365:7: runtime error: null pointer passed as argument 2, which is declared to never be null
third-party-buck/build/glibc/include/string.h:70:33: note: nonnull attribute specified here
    #0 0x433a39 in _ZZN5folly13fbstring_coreIcEC1EPKcmbENKUlvE_clEv ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////buck-out/gen/folly/__default_headers__#default,headers/folly/FBString.h:365:7
    #1 0x4335a9 in _ZN5folly14ScopeGuardImplIZNS_13fbstring_coreIcEC1EPKcmbEUlvE_E7executeEv ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////buck-out/gen/folly/__default_headers__#default,headers/folly/ScopeGuard.h:153:29
    #2 0x4335a9 in _ZN5folly14ScopeGuardImplIZNS_13fbstring_coreIcEC1EPKcmbEUlvE_ED2Ev ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////buck-out/gen/folly/__default_headers__#default,headers/folly/ScopeGuard.h:130
    #3 0x4335a9 in folly::fbstring_core<char>::fbstring_core(char const*, unsigned long, bool) ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////buck-out/gen/folly/__default_headers__#default,headers/folly/FBString.h:427
    #4 0x4353fa in folly::basic_fbstring<char, std::char_traits<char>, std::allocator<char>, folly::fbstring_core<char> >::basic_fbstring(char const*, unsigned long, std::allocator<char> const&) ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////buck-out/gen/folly/__default_headers__#default,headers/folly/FBString.h:1038:9
```

Reviewed By: Gownta

Differential Revision: D3321266

fbshipit-source-id: 28d5aef16e91a98066a1de6bab95403fbc63eaab
ghost pushed a commit that referenced this pull request May 30, 2016
@Orvid
Switch uses of <sys/file.h> to <folly/portability/SysFile.h>
64ed63b 
Summary: Diff #2 of 14.

Reviewed By: mzlee

Differential Revision: D3115972

fbshipit-source-id: c75cbc95f37e8f2c2bf81311a8ca08ee766fc107
@Bklyn Bklyn mentioned this pull request Jul 18, 2016
Closed
@tian-yuan tian-yuan mentioned this pull request Jul 23, 2016
Closed
@tdauth tdauth mentioned this pull request Mar 3, 2017
Closed
@abhiguru abhiguru mentioned this pull request Mar 23, 2017
Closed
facebook-github-bot pushed a commit that referenced this pull request Mar 31, 2017
@wez @facebook-github-bot
avoid UB in StringKeyedCommon for default constructed StringPiece
f1a4f40 
Summary:
ubsan flags this problem:

```
buck-out/opt-ubsan/gen/folly/__default_headers__#header-mode-symlink-tree-with-header-map,headers/folly/experimental/StringKeyedCommon.h:31:18: runtime error: null pointer passed as argument 2, which is declared to never be null
third-party-buck/gcc-4.9-glibc-2.20-fb/build/glibc/include/string.h:47:28: note: nonnull attribute specified here
    #0 0x5cb88f in std::pair<std::__detail::_Node_iterator<std::pair<folly::Range<char const*> const, facebook::eden::GitIgnore>, false, true>, bool> folly::StringKeyedUnorderedMap<facebook::eden::GitIgnore, folly::Hash, std::equal_to<folly::Range<char const*> >, std::allocator<std::pair<folly::Range<char const*> const, facebook::eden::GitIgnore> > >::emplace<facebook::eden::GitIgnore>(folly::Range<char const*>, facebook::eden::GitIgnore&&) buck-out/opt-ubsan/gen/folly/__default_headers__#header-mode-symlink-tree-with-header-map,headers/folly/experimental/StringKeyedCommon.h:31
    #1 0x5c6652 in facebook::eden::(anonymous namespace)::IgnoreChecker::isIgnored(facebook::eden::detail::RelativePathBase<folly::Range<char const*> >) eden/fs/inodes/Dirstate.cpp:226
    #2 0x5c6037 in facebook::eden::(anonymous namespace)::IgnoreChecker::isIgnored(facebook::eden::detail::RelativePathBase<folly::Range<char const*> >) eden/fs/inodes/Dirstate.cpp:180
    #3 0x5c5173 in facebook::eden::Dirstate::getStatusForExistingDirectory(facebook::eden::detail::RelativePathBase<folly::Range<char const*> >) const eden/fs/inodes/Dirstate.cpp:372
    #4 0x5c4509 in facebook::eden::Dirstate::getStatus() const eden/fs/inodes/Dirstate.cpp:272
    #5 0x50a448 in verifyExpectedDirstate(facebook::eden::Dirstate const*, std::unordered_map<std::basic_fbstring<char, std::char_traits<char>, std::allocator<char>, std::fbstring_core<char> >, facebook::eden::StatusCode, std::hash<std::basic_fbstring<char, std::char_traits<char>, std::allocator<char>, std::fbstring_core<char> > >, std::equal_to<std::basic_fbstring<char, std::char_traits<char>, std::allocator<char>, std::fbstring_core<char> > >, std::allocator<std::pair<std::basic_fbstring<char, std::char_traits<char>, std::allocator<char>, std::fbstring_core<char> > const, facebook::eden::StatusCode> > >&&) eden/fs/inodes/test/DirstateTest.cpp:48
    #6 0x50c70c in Dirstate_addDirectoriesWithMixOfFiles_Test::TestBody() eden/fs/inodes/test/DirstateTest.cpp:220
    #7 0xb45857 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/engshare/third-party2/gtest/1.7.0/src/gtest-1.7.0/./src/gtest.cc:2364
    #8 0xb36784 in testing::Test::Run() /home/engshare/third-party2/gtest/1.7.0/src/gtest-1.7.0/./src/gtest.cc:2437
    #9 0xb36957 in testing::TestInfo::Run() [clone .part.558] /home/engshare/third-party2/gtest/1.7.0/src/gtest-1.7.0/./src/gtest.cc:2612
    #10 0xb36b74 in testing::TestCase::Run() [clone .part.559] /home/engshare/third-party2/gtest/1.7.0/src/gtest-1.7.0/./src/gtest.cc:2587
    #11 0xb3806e in testing::internal::UnitTestImpl::RunAllTests() [clone .part.561] /home/engshare/third-party2/gtest/1.7.0/src/gtest-1.7.0/./src/gtest.cc:4571
    #12 0xb382d9 in testing::UnitTest::Run() /home/engshare/third-party2/gtest/1.7.0/src/gtest-1.7.0/./src/gtest.cc:4519
    #13 0x861ca7 in main third-party-buck/gcc-4.9-glibc-2.20-fb/build/gtest/include/gtest/gtest.h:2326
    #14 0x7f20be0740f5 in __libc_start_main /home/engshare/third-party2/glibc/2.20/src/glibc-2.20/csu/libc-start.c:289
    #15 0x4a552c in _start /home/engshare/third-party2/glibc/2.20/src/glibc-2.20/csu/../sysdeps/x86_64/start.S:122

UndefinedBehaviorSanitizer: undefined-behavior buck-out/opt-ubsan/gen/folly/__default_headers__#header-mode-symlink-tree-with-header-map,headers/folly/experimental/StringKeyedCommon.h:31:18
```

The issue is that `StringPiece` default constructs to `{nullptr, nullptr}` as a valid representation of an empty string.
This is tripping up UBSAN in this case.

The fix is a trivial nullptr check

Reviewed By: igorsugak

Differential Revision: D4791015

fbshipit-source-id: dec7484b29ecb29c17b8dd6a9b0e8093f07d63cb
@abhiguru abhiguru mentioned this pull request Apr 24, 2017
Closed
facebook-github-bot pushed a commit that referenced this pull request Jun 1, 2017
@ddrcoder @facebook-github-bot
Fix UB in folly/experimental/Bits.h
a1d6269 
Summary:
Test output before the fix is applied:
```
folly/experimental/Bits.h:247:59: runtime error: signed integer overflow: -2147483648 - 1 cannot be represented in type 'int'
     #0 folly/experimental/Bits.h:247 folly::Bits<...>::set(int*, unsigned long, unsigned long, int)
     #1 folly/experimental/test/BitsTest.cpp:228 std::enable_if<...>::type (anonymous namespace)::testSet<...>(unsigned char*, unsigned long, unsigned long, int)
     #2 folly/experimental/test/BitsTest.cpp:263 Bits_Boundaries_Test::TestBody()
    #17 folly/experimental/test/BitsTest.cpp:381 main
```

Reviewed By: philippv

Differential Revision: D5160789

fbshipit-source-id: 43f1926d58f1a5c019d4f8794d10a7a80a5c4749
@sanjosh sanjosh mentioned this pull request Aug 11, 2017
Closed
@D-PorcoRosso D-PorcoRosso mentioned this pull request Aug 29, 2017
Closed
facebook-github-bot pushed a commit that referenced this pull request Oct 26, 2017
@kennyyu @facebook-github-bot
folly: Fix data race in folly::Codel
9fb1f36 
Summary:
Data race reported by TSAN:

```
WARNING: ThreadSanitizer: data race (pid=608219)
  Read of size 1 at 0x7b5800000c29 by thread T314:
    #0 0x60b3441 in folly::Codel::overloaded(std::chrono::duration<long, std::ratio<1l, 1000000000l> >) ./folly/executors/Codel.cpp:76
    #1 0x5c1222 in apache::thrift::concurrency::ThreadManager::ImplT<folly::LifoSemImpl<std::atomic, folly::Baton<std::atomic, true, true> > >::Worker<folly::LifoSemImpl<std::atomic, folly::Baton<std::atomic, true, true> > >::run() ./thrift/lib/cpp/concurrency/ThreadManager.tcc:119
    #2 0x5d803e7 in apache::thrift::concurrency::PthreadThread::threadMain(void*) ./thrift/lib/cpp/concurrency/PosixThreadFactory.cpp:200
    #3 0x619739d in __tsan_thread_start_func crtstuff.c:?

  Previous write of size 1 at 0x7b5800000c29 by thread T315:
    #0 0x60b33e4 in folly::Codel::overloaded(std::chrono::duration<long, std::ratio<1l, 1000000000l> >) ??:?
    #1 0x5c1222 in apache::thrift::concurrency::ThreadManager::ImplT<folly::LifoSemImpl<std::atomic, folly::Baton<std::atomic, true, true> > >::Worker<folly::LifoSemImpl<std::atomic, folly::Baton<std::atomic, true, true> > >::run() ./thrift/lib/cpp/concurrency/ThreadManager.tcc:119
    #2 0x5d803e7 in apache::thrift::concurrency::PthreadThread::threadMain(void*) ./thrift/lib/cpp/concurrency/PosixThreadFactory.cpp:200
    #3 0x619739d in __tsan_thread_start_func crtstuff.c:?

  Location is heap block of size 768 at 0x7b5800000c00 allocated by main thread:
    #0 0x616ab83 in operator new(unsigned long) ??:?
    #1 0x53cb92 in __gnu_cxx::new_allocator<std::_Sp_counted_ptr_inplace<apache::thrift::concurrency::SimpleThreadManager<folly::LifoSemImpl<std::atomic, folly::Baton<std::atomic, true, true> > >, std::allocator<apache::thrift::concurrency::SimpleThreadManager<folly::LifoSemImpl<std::atomic, folly::Baton<std::atomic, true, true> > > >, (__gnu_cxx::_Lock_policy)2> >::allocate(unsigned long, void const*)
    ...
```

It looks like there are multiple threads reading and writing the `overloaded_` bool. To fix it, wrap it in a `std::atomic`.

Reviewed By: yfeldblum, meyering

Differential Revision: D6149766

fbshipit-source-id: 605b29fa2f602d2ed2dfc22e46b739ef169f914e
@PhoebeHui PhoebeHui mentioned this pull request Mar 14, 2018
Closed
facebook-github-bot pushed a commit that referenced this pull request Oct 4, 2018
@kennyyu @facebook-github-bot
Fix TSAN lock inversion with folly::RequestContext::setContext
b2a3305 
Summary:
TSAN flagged the following lock inversion below. The reason for this lock inversion
is because of the following example usage:

1. set request context B, save old one A (lock B, lock A)
2. later on, reset request context back to A, get the old one back B (lock A, lock B)

In the two calls, we always locked the the new context first, resulting
in a lock inversion. The fix is to always acquire them in a consistent
order using `folly::acquireLocked`

TSAN report:

```
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=1574307)
  Cycle in lock order graph: M159 (0x7b1400000138) => M157 (0x7b14000000e8) => M159

  Mutex M157 acquired here while holding mutex M159 in main thread:
    #0 0x48f18c in AnnotateRWLockAcquired ??:?
    #1 0x315495 in folly::SharedMutexImpl<false, void, std::atomic, false>::lock_shared() ./folly/SharedMutex.h:385
    #2 0x1b0eb8 in folly::detail::LockTraitsImpl<folly::SharedMutexImpl<false, void, std::atomic, false>, (folly::detail::MutexLevel)1, false>::lock_shared(folly::SharedMutexImpl<false, void, std::atomic, false>&) ./folly/LockTraits.h:157
    #3 0x1b0e68 in std::integral_constant<bool, true> folly::LockPolicyShared::lock<folly::SharedMutexImpl<false, void, std::atomic, false> >(folly::SharedMutexImpl<false, void, std::atomic, false>&) ./folly/LockTraits.h:493
    #4 0x1d63a9 in LockedPtrBase ./folly/Synchronized.h:1026
    #5 0x1d6258 in LockedPtr ./folly/Synchronized.h:1326
    #6 0x1c9a17 in folly::SynchronizedBase<folly::Synchronized<folly::RequestContext::State, folly::SharedMutexImpl<false, void, std::atomic, false> >, (folly::detail::MutexLevel)1>::rlock() const ./folly/Synchronized.h:123
    #7 0x1c7530 in folly::RequestContext::setContext(std::shared_ptr<folly::RequestContext>) ./folly/io/async/Request.cpp:243
    #8 0x36a7b in folly::RequestContext::create() ./folly/io/async/Request.h:119
    #9 0x34916 in RequestContextScopeGuard ./folly/io/async/Request.h:269

  Mutex M159 previously acquired by the same thread here:
    #0 0x48f18c in AnnotateRWLockAcquired ??:?
    #1 0x315495 in folly::SharedMutexImpl<false, void, std::atomic, false>::lock_shared() ./folly/SharedMutex.h:385
    #2 0x1b0eb8 in folly::detail::LockTraitsImpl<folly::SharedMutexImpl<false, void, std::atomic, false>, (folly::detail::MutexLevel)1, false>::lock_shared(folly::SharedMutexImpl<false, void, std::atomic, false>&) ./folly/LockTraits.h:157
    #3 0x1b0e68 in std::integral_constant<bool, true> folly::LockPolicyShared::lock<folly::SharedMutexImpl<false, void, std::atomic, false> >(folly::SharedMutexImpl<false, void, std::atomic, false>&) ./folly/LockTraits.h:493
    #4 0x1d63a9 in LockedPtrBase ./folly/Synchronized.h:1026
    #5 0x1d6258 in LockedPtr ./folly/Synchronized.h:1326
    #6 0x1c9a17 in folly::SynchronizedBase<folly::Synchronized<folly::RequestContext::State, folly::SharedMutexImpl<false, void, std::atomic, false> >, (folly::detail::MutexLevel)1>::rlock() const ./folly/Synchronized.h:123
    #7 0x1c7516 in folly::RequestContext::setContext(std::shared_ptr<folly::RequestContext>) ./folly/io/async/Request.cpp:242
    #8 0x36a7b in folly::RequestContext::create() ./folly/io/async/Request.h:119
    #9 0x34916 in RequestContextScopeGuard ./folly/io/async/Request.h:269

  Mutex M159 acquired here while holding mutex M157 in main thread:
    #0 0x48f18c in AnnotateRWLockAcquired ??:?
    #1 0x315495 in folly::SharedMutexImpl<false, void, std::atomic, false>::lock_shared() ./folly/SharedMutex.h:385
    #2 0x1b0eb8 in folly::detail::LockTraitsImpl<folly::SharedMutexImpl<false, void, std::atomic, false>, (folly::detail::MutexLevel)1, false>::lock_shared(folly::SharedMutexImpl<false, void, std::atomic, false>&) ./folly/LockTraits.h:157
    #3 0x1b0e68 in std::integral_constant<bool, true> folly::LockPolicyShared::lock<folly::SharedMutexImpl<false, void, std::atomic, false> >(folly::SharedMutexImpl<false, void, std::atomic, false>&) ./folly/LockTraits.h:493
    #4 0x1d63a9 in LockedPtrBase ./folly/Synchronized.h:1026
    #5 0x1d6258 in LockedPtr ./folly/Synchronized.h:1326
    #6 0x1c9a17 in folly::SynchronizedBase<folly::Synchronized<folly::RequestContext::State, folly::SharedMutexImpl<false, void, std::atomic, false> >, (folly::detail::MutexLevel)1>::rlock() const ./folly/Synchronized.h:123
    #7 0x1c7530 in folly::RequestContext::setContext(std::shared_ptr<folly::RequestContext>) ./folly/io/async/Request.cpp:243
    #8 0x34b99 in ~RequestContextScopeGuard ./folly/io/async/Request.h:278

  Mutex M157 previously acquired by the same thread here:
    #0 0x48f18c in AnnotateRWLockAcquired ??:?
    #1 0x315495 in folly::SharedMutexImpl<false, void, std::atomic, false>::lock_shared() ./folly/SharedMutex.h:385
    #2 0x1b0eb8 in folly::detail::LockTraitsImpl<folly::SharedMutexImpl<false, void, std::atomic, false>, (folly::detail::MutexLevel)1, false>::lock_shared(folly::SharedMutexImpl<false, void, std::atomic, false>&) ./folly/LockTraits.h:157
    #3 0x1b0e68 in std::integral_constant<bool, true> folly::LockPolicyShared::lock<folly::SharedMutexImpl<false, void, std::atomic, false> >(folly::SharedMutexImpl<false, void, std::atomic, false>&) ./folly/LockTraits.h:493
    #4 0x1d63a9 in LockedPtrBase ./folly/Synchronized.h:1026
    #5 0x1d6258 in LockedPtr ./folly/Synchronized.h:1326
    #6 0x1c9a17 in folly::SynchronizedBase<folly::Synchronized<folly::RequestContext::State, folly::SharedMutexImpl<false, void, std::atomic, false> >, (folly::detail::MutexLevel)1>::rlock() const ./folly/Synchronized.h:123
    #7 0x1c7516 in folly::RequestContext::setContext(std::shared_ptr<folly::RequestContext>) ./folly/io/async/Request.cpp:242
    #8 0x34b99 in ~RequestContextScopeGuard ./folly/io/async/Request.h:278

```

Reviewed By: igorsugak

Differential Revision: D9797715

fbshipit-source-id: a0f991ae0ec8f3e9d33af6e05db49eceaf3a5174
sandraiser pushed a commit to sandraiser/folly that referenced this pull request Mar 4, 2019
@kennyyu @sandraiser
Fix TSAN lock inversion with folly::RequestContext::setContext
d528696 
Summary:
TSAN flagged the following lock inversion below. The reason for this lock inversion
is because of the following example usage:

1. set request context B, save old one A (lock B, lock A)
2. later on, reset request context back to A, get the old one back B (lock A, lock B)

In the two calls, we always locked the the new context first, resulting
in a lock inversion. The fix is to always acquire them in a consistent
order using `folly::acquireLocked`

TSAN report:

```
WARNING: ThreadSanitizer: lock-order-inversion (potential deadlock) (pid=1574307)
  Cycle in lock order graph: M159 (0x7b1400000138) => M157 (0x7b14000000e8) => M159

  Mutex M157 acquired here while holding mutex M159 in main thread:
    #0 0x48f18c in AnnotateRWLockAcquired ??:?
    #1 0x315495 in folly::SharedMutexImpl<false, void, std::atomic, false>::lock_shared() ./folly/SharedMutex.h:385
    facebook#2 0x1b0eb8 in folly::detail::LockTraitsImpl<folly::SharedMutexImpl<false, void, std::atomic, false>, (folly::detail::MutexLevel)1, false>::lock_shared(folly::SharedMutexImpl<false, void, std::atomic, false>&) ./folly/LockTraits.h:157
    facebook#3 0x1b0e68 in std::integral_constant<bool, true> folly::LockPolicyShared::lock<folly::SharedMutexImpl<false, void, std::atomic, false> >(folly::SharedMutexImpl<false, void, std::atomic, false>&) ./folly/LockTraits.h:493
    facebook#4 0x1d63a9 in LockedPtrBase ./folly/Synchronized.h:1026
    facebook#5 0x1d6258 in LockedPtr ./folly/Synchronized.h:1326
    facebook#6 0x1c9a17 in folly::SynchronizedBase<folly::Synchronized<folly::RequestContext::State, folly::SharedMutexImpl<false, void, std::atomic, false> >, (folly::detail::MutexLevel)1>::rlock() const ./folly/Synchronized.h:123
    facebook#7 0x1c7530 in folly::RequestContext::setContext(std::shared_ptr<folly::RequestContext>) ./folly/io/async/Request.cpp:243
    facebook#8 0x36a7b in folly::RequestContext::create() ./folly/io/async/Request.h:119
    facebook#9 0x34916 in RequestContextScopeGuard ./folly/io/async/Request.h:269

  Mutex M159 previously acquired by the same thread here:
    #0 0x48f18c in AnnotateRWLockAcquired ??:?
    #1 0x315495 in folly::SharedMutexImpl<false, void, std::atomic, false>::lock_shared() ./folly/SharedMutex.h:385
    facebook#2 0x1b0eb8 in folly::detail::LockTraitsImpl<folly::SharedMutexImpl<false, void, std::atomic, false>, (folly::detail::MutexLevel)1, false>::lock_shared(folly::SharedMutexImpl<false, void, std::atomic, false>&) ./folly/LockTraits.h:157
    facebook#3 0x1b0e68 in std::integral_constant<bool, true> folly::LockPolicyShared::lock<folly::SharedMutexImpl<false, void, std::atomic, false> >(folly::SharedMutexImpl<false, void, std::atomic, false>&) ./folly/LockTraits.h:493
    facebook#4 0x1d63a9 in LockedPtrBase ./folly/Synchronized.h:1026
    facebook#5 0x1d6258 in LockedPtr ./folly/Synchronized.h:1326
    facebook#6 0x1c9a17 in folly::SynchronizedBase<folly::Synchronized<folly::RequestContext::State, folly::SharedMutexImpl<false, void, std::atomic, false> >, (folly::detail::MutexLevel)1>::rlock() const ./folly/Synchronized.h:123
    facebook#7 0x1c7516 in folly::RequestContext::setContext(std::shared_ptr<folly::RequestContext>) ./folly/io/async/Request.cpp:242
    facebook#8 0x36a7b in folly::RequestContext::create() ./folly/io/async/Request.h:119
    facebook#9 0x34916 in RequestContextScopeGuard ./folly/io/async/Request.h:269

  Mutex M159 acquired here while holding mutex M157 in main thread:
    #0 0x48f18c in AnnotateRWLockAcquired ??:?
    #1 0x315495 in folly::SharedMutexImpl<false, void, std::atomic, false>::lock_shared() ./folly/SharedMutex.h:385
    facebook#2 0x1b0eb8 in folly::detail::LockTraitsImpl<folly::SharedMutexImpl<false, void, std::atomic, false>, (folly::detail::MutexLevel)1, false>::lock_shared(folly::SharedMutexImpl<false, void, std::atomic, false>&) ./folly/LockTraits.h:157
    facebook#3 0x1b0e68 in std::integral_constant<bool, true> folly::LockPolicyShared::lock<folly::SharedMutexImpl<false, void, std::atomic, false> >(folly::SharedMutexImpl<false, void, std::atomic, false>&) ./folly/LockTraits.h:493
    facebook#4 0x1d63a9 in LockedPtrBase ./folly/Synchronized.h:1026
    facebook#5 0x1d6258 in LockedPtr ./folly/Synchronized.h:1326
    facebook#6 0x1c9a17 in folly::SynchronizedBase<folly::Synchronized<folly::RequestContext::State, folly::SharedMutexImpl<false, void, std::atomic, false> >, (folly::detail::MutexLevel)1>::rlock() const ./folly/Synchronized.h:123
    facebook#7 0x1c7530 in folly::RequestContext::setContext(std::shared_ptr<folly::RequestContext>) ./folly/io/async/Request.cpp:243
    facebook#8 0x34b99 in ~RequestContextScopeGuard ./folly/io/async/Request.h:278

  Mutex M157 previously acquired by the same thread here:
    #0 0x48f18c in AnnotateRWLockAcquired ??:?
    #1 0x315495 in folly::SharedMutexImpl<false, void, std::atomic, false>::lock_shared() ./folly/SharedMutex.h:385
    facebook#2 0x1b0eb8 in folly::detail::LockTraitsImpl<folly::SharedMutexImpl<false, void, std::atomic, false>, (folly::detail::MutexLevel)1, false>::lock_shared(folly::SharedMutexImpl<false, void, std::atomic, false>&) ./folly/LockTraits.h:157
    facebook#3 0x1b0e68 in std::integral_constant<bool, true> folly::LockPolicyShared::lock<folly::SharedMutexImpl<false, void, std::atomic, false> >(folly::SharedMutexImpl<false, void, std::atomic, false>&) ./folly/LockTraits.h:493
    facebook#4 0x1d63a9 in LockedPtrBase ./folly/Synchronized.h:1026
    facebook#5 0x1d6258 in LockedPtr ./folly/Synchronized.h:1326
    facebook#6 0x1c9a17 in folly::SynchronizedBase<folly::Synchronized<folly::RequestContext::State, folly::SharedMutexImpl<false, void, std::atomic, false> >, (folly::detail::MutexLevel)1>::rlock() const ./folly/Synchronized.h:123
    facebook#7 0x1c7516 in folly::RequestContext::setContext(std::shared_ptr<folly::RequestContext>) ./folly/io/async/Request.cpp:242
    facebook#8 0x34b99 in ~RequestContextScopeGuard ./folly/io/async/Request.h:278

```

Reviewed By: igorsugak

Differential Revision: D9797715

fbshipit-source-id: a0f991ae0ec8f3e9d33af6e05db49eceaf3a5174
facebook-github-bot pushed a commit that referenced this pull request Apr 10, 2019
@Orvid @facebook-github-bot
AsyncServerSocket::AcceptCallback::connectionAccepted to NetworkSocke…
3c6096c 
…t (attempt #2)

Summary:
The file descriptor API here needs to go away, so switch this API to NetworkSocket

It is expected that this commit will cause a number of Open Source projects to temporarily show up as broken. This is due to the fact that not all projects get synced to Github at the exact same time, so the builds may temporarily be fetching an older version of it's dependencies than it needs to :) It should fix itself quickly.

Reviewed By: yfeldblum

Differential Revision: D14673328

fbshipit-source-id: c5842fa5dc383d50043e0d8228e35d03b10a1c6b
This was referenced Jul 12, 2019
Closed
Closed
@tabokie tabokie mentioned this pull request Dec 4, 2019
Closed
facebook-github-bot pushed a commit that referenced this pull request Dec 18, 2019
@facebook-github-bot
fix unsigned-integer-overflow
c4a9979 
Summary:
Exposed by UBSAN's unsigned-integer-overflow:
```
> folly/experimental/observer/detail/Core.cpp:179:9: runtime error: unsigned integer overflow: 0 - 1 cannot be represented in type 'size_t' (aka 'unsigned long')
>     #0 0x7f605837dfea in folly::observer_detail::Core::removeStaleDependents()::$_4::operator()(std::vector<std::weak_ptr<folly::observer_detail::Core>, std::allocator<std::weak_ptr<folly::observer_detail::Core> > >&) const folly/experimental/observer/detail/Core.cpp:179
>     #1 0x7f6058344b0c in auto folly::SynchronizedBase<folly::Synchronized<std::vector<std::weak_ptr<folly::observer_detail::Core>, std::allocator<std::weak_ptr<folly::observer_detail::Core> > >, folly::SharedMutexImpl<false, void, std::atomic, false, false> >, (folly::detail::MutexLevel)1>::withWLock<folly::observer_detail::Core::removeStaleDependents()::$_4>(folly::observer_detail::Core::removeStaleDependents()::$_4&&) folly/Synchronized.h:210
>     #2 0x7f6058344917 in folly::observer_detail::Core::removeStaleDependents() folly/experimental/observer/detail/Core.cpp:174
>     #3 0x7f60583785cb in folly::observer_detail::Core::~Core()::$_2::operator()(std::unordered_set<std::shared_ptr<folly::observer_detail::Core>, std::hash<std::shared_ptr<folly::observer_detail::Core> >, std::equal_to<std::shared_ptr<folly::observer_detail::Core> >, std::allocator<std::shared_ptr<folly::observer_detail::Core> > > const&) const folly/experimental/observer/detail/Core.cpp:156
>     #4 0x7f6058343cdc in auto folly::SynchronizedBase<folly::Synchronized<std::unordered_set<std::shared_ptr<folly::observer_detail::Core>, std::hash<std::shared_ptr<folly::observer_detail::Core> >, std::equal_to<std::shared_ptr<folly::observer_detail::Core> >, std::allocator<std::shared_ptr<folly::observer_detail::Core> > >, folly::SharedMutexImpl<false, void, std::atomic, false, false> >, (folly::detail::MutexLevel)1>::withWLock<folly::observer_detail::Core::~Core()::$_2>(folly::ob
server_detail::Core::~Core()::$_2&&) folly/Synchronized.h:210
>     #5 0x7f6058343a17 in folly::observer_detail::Core::~Core() folly/experimental/observer/detail/Core.cpp:154
>     #6 0x7f605838b8ca in std::_Sp_counted_ptr<folly::observer_detail::Core*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() buck-out/cells/fbsource/gen/third-party/gcc/7.x/stdc++-headers#header-mode-symlink-tree-with-header-map,headers/bits/shared_ptr_base.h:378
>     #7 0x7f605834d411 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() buck-out/cells/fbsource/gen/third-party/gcc/7.x/stdc++-headers#header-mode-symlink-tree-with-header-map,headers/bits/shared_ptr_base.h:156
>     #8 0x7f605834d332 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() buck-out/cells/fbsource/gen/third-party/gcc/7.x/stdc++-headers#header-mode-symlink-tree-with-header-map,headers/bits/shared_ptr_base.h:686
>     #9 0x7f605834d23a in std::__shared_ptr<folly::observer_detail::Core, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr() buck-out/cells/fbsource/gen/third-party/gcc/7.x/stdc++-headers#header-mode-symlink-tree-with-header-map,headers/bits/shared_ptr_base.h:1125
>     #10 0x7f605833c416 in std::shared_ptr<folly::observer_detail::Core>::~shared_ptr() buck-out/cells/fbsource/gen/third-party/gcc/7.x/stdc++-headers#header-mode-symlink-tree-with-header-map,headers/bits/shared_ptr.h:93
>     #11 0x7f6058376413 in folly::observer_detail::ObserverManager::scheduleRefresh(std::shared_ptr<folly::observer_detail::Core>, unsigned long)::'lambda'()::~() folly/experimental/observer/detail/ObserverManager.h:91
>     #12 0x7f6058377014 in unsigned long folly::detail::function::execSmall<folly::observer_detail::ObserverManager::scheduleRefresh(std::shared_ptr<folly::observer_detail::Core>, unsigned long)::'lambda'()>(folly::detail::function::Op, folly::detail::function::Data*, folly::detail::function::Data*) folly/Function.h:592
>     #13 0x7f6058377788 in folly::Function<void ()>::exec(folly::detail::function::Op, folly::detail::function::Data*, folly::detail::function::Data*) const folly/Function.h:649
>     #14 0x7f6058376380 in folly::Function<void ()>::~Function() folly/Function.h:781
>     #15 0x7f6058477b99 in folly::observer_detail::ObserverManager::UpdatesManager::CurrentQueue::CurrentQueue()::'lambda'()::operator()() const folly/experimental/observer/detail/ObserverManager.cpp:73
>     #16 0x7f6058473de8 in void std::__invoke_impl<void, folly::observer_detail::ObserverManager::UpdatesManager::CurrentQueue::CurrentQueue()::'lambda'()>(std::__invoke_other, folly::observer_detail::ObserverManager::UpdatesManager::CurrentQueue::CurrentQueue()::'lambda'()&&) buck-out/cells/fbsource/gen/third-party/gcc/7.x/stdc++-headers#header-mode-symlink-tree-with-header-map,headers/bits/invoke.h:60
>     #17 0x7f6058473c08 in std::__invoke_result<folly::observer_detail::ObserverManager::UpdatesManager::CurrentQueue::CurrentQueue()::'lambda'()>::type std::__invoke<folly::observer_detail::ObserverManager::UpdatesManager::CurrentQueue::CurrentQueue()::'lambda'()>(folly::observer_detail::ObserverManager::UpdatesManager::CurrentQueue::CurrentQueue()::'lambda'()&&) buck-out/cells/fbsource/gen/third-party/gcc/7.x/stdc++-headers#header-mode-symlink-tree-with-header-map,headers/bits/invok
e.h:95
>     #18 0x7f6058473b66 in decltype(std::__invoke(_S_declval<0ul>())) std::thread::_Invoker<std::tuple<folly::observer_detail::ObserverManager::UpdatesManager::CurrentQueue::CurrentQueue()::'lambda'()> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) buck-out/cells/fbsource/gen/third-party/gcc/7.x/stdc++-headers#header-mode-symlink-tree-with-header-map,headers/thread:234
>     #19 0x7f6058473a43 in std::thread::_Invoker<std::tuple<folly::observer_detail::ObserverManager::UpdatesManager::CurrentQueue::CurrentQueue()::'lambda'()> >::operator()() buck-out/cells/fbsource/gen/third-party/gcc/7.x/stdc++-headers#header-mode-symlink-tree-with-header-map,headers/thread:243
>     #20 0x7f6058473549 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<folly::observer_detail::ObserverManager::UpdatesManager::CurrentQueue::CurrentQueue()::'lambda'()> > >::_M_run() buck-out/cells/fbsource/gen/third-party/gcc/7.x/stdc++-headers#header-mode-symlink-tree-with-header-map,headers/thread:186
>     #21 0x7f60576bac5f in std::execute_native_thread_routine(void*) third-party/gcc/7.x/libstdc++-v3/src/c++11/thread.cc:83
>     #22 0x7f605639e6b5 in start_thread /home/engshare/third-party2/glibc/2.26/src/glibc-2.26/nptl/pthread_create.c:465:7
>     #23 0x7f6055ecdebe in __GI___clone /home/engshare/third-party2/glibc/2.26/src/glibc-2.26/sysdeps/unix/sysv/linux/x86_64/clone.S:95
>
> SUMMARY: UndefinedBehaviorSanitizer: unsigned-integer-overflow folly/experimental/observer/detail/Core.cpp:179:9
```

Fix the error by rearranging increment and decrement operations.

Reviewed By: leikahing

Differential Revision: D19164999

fbshipit-source-id: d3c9b638492d0bf44c2acaac1e95fafe8b785182
facebook-github-bot pushed a commit that referenced this pull request Dec 19, 2019
@facebook-github-bot
fix misaligned-pointer-use in StlVectorTest
be4b879 
Summary:
Exposed by UBSAN's misaligned-pointer-use:
```lang=bash
> folly/test/stl_tests/StlVectorTest.cpp:544:12: runtime error: upcast of misaligned address 0x0000feebdaed for type 'DataTracker<false>', which requires 8 byte alignment
> 0x0000feebdaed: note: pointer points here
> <memory cannot be printed>
>     #0 0x6c9b93 in DataTracker<false>::~DataTracker() folly/test/stl_tests/StlVectorTest.cpp:544
>     #1 0x6c94fe in Data<0u, 0ul>::~Data() folly/test/stl_tests/StlVectorTest.cpp:605
>     #2 0xcc5f93 in void test_iteratorInsertionN2<folly::fbvector<Data<0u, 0ul>, std::allocator<Data<0u, 0ul> > > >(std::integral_constant<bool, true>) folly/test/stl_tests/StlVectorTest.cpp:2422
>     #3 0x5c9bac in void test_iteratorInsertionN3<folly::fbvector<Data<0u, 0ul>, std::allocator<Data<0u, 0ul> > > >() folly/test/stl_tests/StlVectorTest.cpp:2422
>     #4 0x5c67bd in FBVector_iteratorInsertionN_Test::TestBody() folly/test/stl_tests/StlVectorTest.cpp:2422
>     #5 0x7f9c4e5316c9 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) xplat/third-party/gmock/googletest-1.8.0/googletest/src/gtest.cc:2464
>     #6 0x7f9c4e530dcc in testing::Test::Run() xplat/third-party/gmock/googletest-1.8.0/googletest/src/gtest.cc:2480
>     #7 0x7f9c4e5350a0 in testing::TestInfo::Run() xplat/third-party/gmock/googletest-1.8.0/googletest/src/gtest.cc:2662
>     #8 0x7f9c4e539408 in testing::TestCase::Run() xplat/third-party/gmock/googletest-1.8.0/googletest/src/gtest.cc:2780
>     #9 0x7f9c4e55a672 in testing::internal::UnitTestImpl::RunAllTests() xplat/third-party/gmock/googletest-1.8.0/googletest/src/gtest.cc:4655
>     #10 0x7f9c4e5595de in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) xplat/third-party/gmock/googletest-1.8.0/googletest/src/gtest.cc:2464
>     #11 0x7f9c4e558b72 in testing::UnitTest::Run() xplat/third-party/gmock/googletest-1.8.0/googletest/src/gtest.cc:4263
>     #12 0x6591f3 in RUN_ALL_TESTS() buck-out/cells/fbsource/gen/xplat/third-party/gmock/gtest_headers#header-mode-symlink-tree-with-header-map,headers/gtest/gtest.h:2247
>     #13 0x659139 in main folly/test/stl_tests/StlVectorTest.cpp:3074
>     #14 0x7f9c4c75b1a5 in __libc_start_main /home/engshare/third-party2/glibc/2.26/src/glibc-2.26/csu/libc-start.c:308:16
>     #15 0x4de029 in _start /home/engshare/third-party2/glibc/2.26/src/glibc-2.26/sysdeps/x86_64/start.S:120
>
> SUMMARY: UndefinedBehaviorSanitizer: misaligned-pointer-use folly/test/stl_tests/StlVectorTest.cpp:544:12
```

Make the crafted pointer address aligned.

Reviewed By: Gownta

Differential Revision: D19166262

fbshipit-source-id: f766656d031079350838598b135275bdbc047642
facebook-github-bot pushed a commit that referenced this pull request Jan 15, 2020
@facebook-github-bot
prevent asan failures in SingletonThreadLocal
9450078 
Summary:
I have been encountering

```
AddressSanitizer:DEADLYSIGNAL
=================================================================
==937752==ERROR: AddressSanitizer: SEGV on unknown address 0x24a12000e4f18 (pc 0x7f3902a6e696 bp 0x7f37bc5e4380 sp 0x7f37bc5e42a0 T107)
==937752==The signal is caused by a READ memory access.
SCARINESS: 20 (wild-addr-read)

    #0 0x7f3902a6e695 in folly::ThreadLocalPtr<folly::SingletonThreadLocal<folly::hazptr_tc<std::atomic>, void, folly::detail::DefaultMake<folly::hazptr_tc<std::atomic> >, void>::Wrapper, void, void>::get() const folly/ThreadLocal.h:160
    #1 0x7f3902a6d600 in get folly/ThreadLocal.h:69
    #2 0x7f3902a6d600 in folly::ThreadLocal<folly::SingletonThreadLocal<folly::hazptr_tc<std::atomic>, void, folly::detail::DefaultMake<folly::hazptr_tc<std::atomic> >, void>::Wrapper, void, void>::operator*() const folly/ThreadLocal.h:78
    #3 0x7f3902a6ca1b in folly::SingletonThreadLocal<folly::hazptr_tc<std::atomic>, void, folly::detail::DefaultMake<folly::hazptr_tc<std::atomic> >, void>::getWrapper() folly/SingletonThreadLocal.h:149
    #4 0x7f3902a6cbe5 in folly::SingletonThreadLocal<folly::hazptr_tc<std::atomic>, void, folly::detail::DefaultMake<folly::hazptr_tc<std::atomic> >, void>::LocalLifetime::~LocalLifetime() folly/SingletonThreadLocal.h:121
    #5 0x7f38f04d9fd5 in (anonymous namespace)::run(void*) /data/users/kirkshoop/gcc/trunk/libstdc++-v3/libsupc++/atexit_thread.cc:75:16
    #6 0x7f38ef9ee551 in __nptl_deallocate_tsd.part.8 /home/engshare/third-party2/glibc/2.26/src/glibc-2.26/nptl/pthread_create.c:301:8
    #7 0x7f38ef9ef7c8 in __nptl_deallocate_tsd /home/engshare/third-party2/glibc/2.26/src/glibc-2.26/sysdeps/nptl/futex-internal.h:200:3
    #8 0x7f38ef9ef7c8 in start_thread /home/engshare/third-party2/glibc/2.26/src/glibc-2.26/nptl/pthread_create.c:475:3
```

I tried using a unique Tag for the SingletonThreadLocal for debugging purposes and this seemed to fix the issue.

Reviewed By: magedm

Differential Revision: D19356179

fbshipit-source-id: a18c6b02f0a40e0509753075614eeff71d859bf7
facebook-github-bot pushed a commit that referenced this pull request Jan 23, 2020
@akrieger @facebook-github-bot
Use extended alignment safe allocator in CPUThreadPoolExecutor
d2bc79c 
Summary:
Until C++17 types with extended alignment need special handling
in eg. std::vector and also std::unique_ptr. CPUThreadPoolExecutor has a
default queue that requires extended alignment, but also allows the user
to provide their own blocking queue. To handle this we move the private
member to a shared_ptr which supported type erased destructors, and then
use allocate_shared for the default queue type with a custom allocator
that will satisfy alignment constraints.

```
.../unique_ptr.h:825:34: runtime error: constructor call on misaligned address 0x613000000040 for type 'folly::UnboundedBlockingQueue<folly::CPUThreadPoolExecutor::CPUTask>', which requires 128 byte alignment
0x613000000040: note: pointer points here
 02 00 00 1c  be be be be be be be be  be be be be be be be be  be be be be be be be be  be be be be
              ^
    #0 0x75b35e in std::_MakeUniq<folly::UnboundedBlockingQueue<folly::CPUThreadPoolExecutor::CPUTask> >::__single_object std::make_unique<folly::UnboundedBlockingQueue<folly::CPUThreadPoolExecutor::CPUTask> >() .../unique_ptr.h:825:30
    #1 0x75602f in folly::CPUThreadPoolExecutor::CPUThreadPoolExecutor(unsigned long, std::shared_ptr<folly::ThreadFactory>) xplat/folly/executors/CPUThreadPoolExecutor.cpp:66:18
    #2 0x756c6c in folly::CPUThreadPoolExecutor::CPUThreadPoolExecutor(unsigned long) xplat/folly/executors/CPUThreadPoolExecutor.cpp:84:7
```

Differential Revision: D19237477

fbshipit-source-id: c88b26e2ca17e168f124ba27c0989f787b1ce4fd
facebook-github-bot pushed a commit that referenced this pull request Jan 23, 2020
@akrieger @facebook-github-bot
Use aligned allocation for hazptr_rec<Atom>
cf917cf 
Summary:
The default allocator for `new` does not apparently respect the extended alignment requirement for `hasptr_rec<Atom>`. We use folly's AlignedSysAllocator for these cases, which will then pass UBSan checks for alignment.

```
xplat/folly/synchronization/HazptrDomain.h:572:20: runtime error: constructor call on misaligned address 0x60c0000004c0 for type 'hazptr_rec<atomic>', which requires 128 byte alignment
0x60c0000004c0: note: pointer points here
 02 00 00 7f  be be be be be be be be  be be be be be be be be  be be be be be be be be  be be be be
              ^
    #0 0x77949e in folly::hazptr_domain<std::atomic>::acquire_new_hprec() xplat/folly/synchronization/HazptrDomain.h:572:16
    #1 0x7791cc in folly::hazptr_domain<std::atomic>::hprec_acquire() xplat/folly/synchronization/HazptrDomain.h:227:35
    #2 0x766a7f in folly::hazptr_holder<std::atomic>::hazptr_holder(folly::hazptr_domain<std::atomic>&) xplat/folly/synchronization/HazptrHolder.h:71:21
    #3 0x766a7f in void folly::UnboundedQueue<folly::CPUThreadPoolExecutor::CPUTask, false, false, false, 6ul, 7ul, std::atomic>::enqueueImpl<folly::CPUThreadPoolExecutor::CPUTask>(folly::CPUThreadPoolExecutor::CPUTask&&) xplat/folly/concurrency/UnboundedQueue.h:374
```

Reviewed By: magedm

Differential Revision: D19237973

fbshipit-source-id: 0edea12bb3f028d21830689d52f7e0290d187ff9
facebook-github-bot pushed a commit that referenced this pull request Feb 1, 2020
@akrieger @facebook-github-bot
Use extended alignment safe allocator in CPUThreadPoolExecutor
d8b4f2f 
Summary:
Until C++17 types with extended alignment need special handling
in eg. std::vector and also std::unique_ptr. CPUThreadPoolExecutor has a
default queue that requires extended alignment, but also allows the user
to provide their own blocking queue. To handle this we move the private
member to a shared_ptr which supported type erased destructors, and then
use allocate_shared for the default queue type with a custom allocator
that will satisfy alignment constraints.

```
.../unique_ptr.h:825:34: runtime error: constructor call on misaligned address 0x613000000040 for type 'folly::UnboundedBlockingQueue<folly::CPUThreadPoolExecutor::CPUTask>', which requires 128 byte alignment
0x613000000040: note: pointer points here
 02 00 00 1c  be be be be be be be be  be be be be be be be be  be be be be be be be be  be be be be
              ^
    #0 0x75b35e in std::_MakeUniq<folly::UnboundedBlockingQueue<folly::CPUThreadPoolExecutor::CPUTask> >::__single_object std::make_unique<folly::UnboundedBlockingQueue<folly::CPUThreadPoolExecutor::CPUTask> >() .../unique_ptr.h:825:30
    #1 0x75602f in folly::CPUThreadPoolExecutor::CPUThreadPoolExecutor(unsigned long, std::shared_ptr<folly::ThreadFactory>) xplat/folly/executors/CPUThreadPoolExecutor.cpp:66:18
    #2 0x756c6c in folly::CPUThreadPoolExecutor::CPUThreadPoolExecutor(unsigned long) xplat/folly/executors/CPUThreadPoolExecutor.cpp:84:7
```

Differential Revision: D19669832

fbshipit-source-id: 7fb60a06dbb6a04edddb6b619af7a8cef4995fd2
facebook-github-bot pushed a commit that referenced this pull request Jul 9, 2020
@marksantaniello @facebook-github-bot
Use sized deallocation in SysAllocator and Arena
0d2bdbf 
Summary:
Use sized-deallocation (`sdallocx`) if possible in `folly::SysAllocator` and `folly::Arena`.

`Arena` has always allocated two types of blocks:
1. Normal (fixed-sized): size is the "goodSize adjusted" `minBlockSize`
2. Large (variable-sized): when #1 is too small

Type #2 makes sized-deallocation tricky -- we need somewhere to remember the allocated sizes.

The old code used a single type `Block` and kept a single list. Here I change to have two types and two lists.  The `LargeBlock` has an additional `allocSize` data member.

This makes the Arena object itself 16B larger, but seems better than adding a 4B `allocSize` to each and every block, regardless of type.

Note that, prior to this change, it was possible to `merge()` arenas with different `minBlockSize`.  This is no longer possible.

Reviewed By: yfeldblum

Differential Revision: D22189916

fbshipit-source-id: e6fba48eaae0b5cc8456b856b02d2cfc71c03834
facebook-github-bot pushed a commit that referenced this pull request Jul 10, 2020
@marksantaniello @facebook-github-bot
Use sized deallocation in SysAllocator and Arena (2nd try)
b99077f 
Summary:
Use sized-deallocation (`sdallocx`) if possible in `folly::SysAllocator` and `folly::Arena`.

`Arena` has always allocated two types of blocks:
1. Normal (fixed-sized): size is the "goodSize adjusted" `minBlockSize`
2. Large (variable-sized): when #1 is too small

Type #2 makes sized-deallocation tricky -- we need somewhere to remember the allocated sizes.

The old code used a single type `Block` and kept a single list. Here I change to have two types and two lists.  The `LargeBlock` has an additional `allocSize` data member.

This makes the Arena object itself 16B larger, but seems better than adding a 4B `allocSize` to each and every block, regardless of type.

Note that, prior to this change, it was possible to `merge()` arenas with different `minBlockSize`.  This is no longer possible.

Reviewed By: davidtgoldblatt

Differential Revision: D22466906

fbshipit-source-id: 719f357a0a1f6cfcda3208391837195c3859ab69
machogallo2 pushed a commit to machogallo2/folly that referenced this pull request Sep 5, 2020
@raihan71
Merge pull request facebook#2 from facebook/master
114a551 
Update from upstream facebook/folly
dotconnor pushed a commit to 5448C8/folly that referenced this pull request Mar 19, 2021
@dotconnor
prevent asan failures in SingletonThreadLocal
5a97039 
Summary:
I have been encountering

```
AddressSanitizer:DEADLYSIGNAL
=================================================================
==937752==ERROR: AddressSanitizer: SEGV on unknown address 0x24a12000e4f18 (pc 0x7f3902a6e696 bp 0x7f37bc5e4380 sp 0x7f37bc5e42a0 T107)
==937752==The signal is caused by a READ memory access.
SCARINESS: 20 (wild-addr-read)

    #0 0x7f3902a6e695 in folly::ThreadLocalPtr<folly::SingletonThreadLocal<folly::hazptr_tc<std::atomic>, void, folly::detail::DefaultMake<folly::hazptr_tc<std::atomic> >, void>::Wrapper, void, void>::get() const folly/ThreadLocal.h:160
    facebook#1 0x7f3902a6d600 in get folly/ThreadLocal.h:69
    facebook#2 0x7f3902a6d600 in folly::ThreadLocal<folly::SingletonThreadLocal<folly::hazptr_tc<std::atomic>, void, folly::detail::DefaultMake<folly::hazptr_tc<std::atomic> >, void>::Wrapper, void, void>::operator*() const folly/ThreadLocal.h:78
    facebook#3 0x7f3902a6ca1b in folly::SingletonThreadLocal<folly::hazptr_tc<std::atomic>, void, folly::detail::DefaultMake<folly::hazptr_tc<std::atomic> >, void>::getWrapper() folly/SingletonThreadLocal.h:149
    facebook#4 0x7f3902a6cbe5 in folly::SingletonThreadLocal<folly::hazptr_tc<std::atomic>, void, folly::detail::DefaultMake<folly::hazptr_tc<std::atomic> >, void>::LocalLifetime::~LocalLifetime() folly/SingletonThreadLocal.h:121
    facebook#5 0x7f38f04d9fd5 in (anonymous namespace)::run(void*) /data/users/kirkshoop/gcc/trunk/libstdc++-v3/libsupc++/atexit_thread.cc:75:16
    facebook#6 0x7f38ef9ee551 in __nptl_deallocate_tsd.part.8 /home/engshare/third-party2/glibc/2.26/src/glibc-2.26/nptl/pthread_create.c:301:8
    facebook#7 0x7f38ef9ef7c8 in __nptl_deallocate_tsd /home/engshare/third-party2/glibc/2.26/src/glibc-2.26/sysdeps/nptl/futex-internal.h:200:3
    facebook#8 0x7f38ef9ef7c8 in start_thread /home/engshare/third-party2/glibc/2.26/src/glibc-2.26/nptl/pthread_create.c:475:3
```

I tried using a unique Tag for the SingletonThreadLocal for debugging purposes and this seemed to fix the issue.

Reviewed By: magedm

Differential Revision: D19356179

fbshipit-source-id: a18c6b02f0a40e0509753075614eeff71d859bf7
dotconnor pushed a commit to 5448C8/folly that referenced this pull request Mar 19, 2021
@akrieger @dotconnor
Use extended alignment safe allocator in CPUThreadPoolExecutor
6302db1 
Summary:
Until C++17 types with extended alignment need special handling
in eg. std::vector and also std::unique_ptr. CPUThreadPoolExecutor has a
default queue that requires extended alignment, but also allows the user
to provide their own blocking queue. To handle this we move the private
member to a shared_ptr which supported type erased destructors, and then
use allocate_shared for the default queue type with a custom allocator
that will satisfy alignment constraints.

```
.../unique_ptr.h:825:34: runtime error: constructor call on misaligned address 0x613000000040 for type 'folly::UnboundedBlockingQueue<folly::CPUThreadPoolExecutor::CPUTask>', which requires 128 byte alignment
0x613000000040: note: pointer points here
 02 00 00 1c  be be be be be be be be  be be be be be be be be  be be be be be be be be  be be be be
              ^
    #0 0x75b35e in std::_MakeUniq<folly::UnboundedBlockingQueue<folly::CPUThreadPoolExecutor::CPUTask> >::__single_object std::make_unique<folly::UnboundedBlockingQueue<folly::CPUThreadPoolExecutor::CPUTask> >() .../unique_ptr.h:825:30
    facebook#1 0x75602f in folly::CPUThreadPoolExecutor::CPUThreadPoolExecutor(unsigned long, std::shared_ptr<folly::ThreadFactory>) xplat/folly/executors/CPUThreadPoolExecutor.cpp:66:18
    facebook#2 0x756c6c in folly::CPUThreadPoolExecutor::CPUThreadPoolExecutor(unsigned long) xplat/folly/executors/CPUThreadPoolExecutor.cpp:84:7
```

Differential Revision: D19237477

fbshipit-source-id: c88b26e2ca17e168f124ba27c0989f787b1ce4fd
dotconnor pushed a commit to 5448C8/folly that referenced this pull request Mar 19, 2021
@akrieger @dotconnor
Use aligned allocation for hazptr_rec<Atom>
1701380 
Summary:
The default allocator for `new` does not apparently respect the extended alignment requirement for `hasptr_rec<Atom>`. We use folly's AlignedSysAllocator for these cases, which will then pass UBSan checks for alignment.

```
xplat/folly/synchronization/HazptrDomain.h:572:20: runtime error: constructor call on misaligned address 0x60c0000004c0 for type 'hazptr_rec<atomic>', which requires 128 byte alignment
0x60c0000004c0: note: pointer points here
 02 00 00 7f  be be be be be be be be  be be be be be be be be  be be be be be be be be  be be be be
              ^
    #0 0x77949e in folly::hazptr_domain<std::atomic>::acquire_new_hprec() xplat/folly/synchronization/HazptrDomain.h:572:16
    facebook#1 0x7791cc in folly::hazptr_domain<std::atomic>::hprec_acquire() xplat/folly/synchronization/HazptrDomain.h:227:35
    facebook#2 0x766a7f in folly::hazptr_holder<std::atomic>::hazptr_holder(folly::hazptr_domain<std::atomic>&) xplat/folly/synchronization/HazptrHolder.h:71:21
    facebook#3 0x766a7f in void folly::UnboundedQueue<folly::CPUThreadPoolExecutor::CPUTask, false, false, false, 6ul, 7ul, std::atomic>::enqueueImpl<folly::CPUThreadPoolExecutor::CPUTask>(folly::CPUThreadPoolExecutor::CPUTask&&) xplat/folly/concurrency/UnboundedQueue.h:374
```

Reviewed By: magedm

Differential Revision: D19237973

fbshipit-source-id: 0edea12bb3f028d21830689d52f7e0290d187ff9
dotconnor pushed a commit to 5448C8/folly that referenced this pull request Mar 19, 2021
@akrieger @dotconnor
Use extended alignment safe allocator in CPUThreadPoolExecutor
91d92a5 
Summary:
Until C++17 types with extended alignment need special handling
in eg. std::vector and also std::unique_ptr. CPUThreadPoolExecutor has a
default queue that requires extended alignment, but also allows the user
to provide their own blocking queue. To handle this we move the private
member to a shared_ptr which supported type erased destructors, and then
use allocate_shared for the default queue type with a custom allocator
that will satisfy alignment constraints.

```
.../unique_ptr.h:825:34: runtime error: constructor call on misaligned address 0x613000000040 for type 'folly::UnboundedBlockingQueue<folly::CPUThreadPoolExecutor::CPUTask>', which requires 128 byte alignment
0x613000000040: note: pointer points here
 02 00 00 1c  be be be be be be be be  be be be be be be be be  be be be be be be be be  be be be be
              ^
    #0 0x75b35e in std::_MakeUniq<folly::UnboundedBlockingQueue<folly::CPUThreadPoolExecutor::CPUTask> >::__single_object std::make_unique<folly::UnboundedBlockingQueue<folly::CPUThreadPoolExecutor::CPUTask> >() .../unique_ptr.h:825:30
    facebook#1 0x75602f in folly::CPUThreadPoolExecutor::CPUThreadPoolExecutor(unsigned long, std::shared_ptr<folly::ThreadFactory>) xplat/folly/executors/CPUThreadPoolExecutor.cpp:66:18
    facebook#2 0x756c6c in folly::CPUThreadPoolExecutor::CPUThreadPoolExecutor(unsigned long) xplat/folly/executors/CPUThreadPoolExecutor.cpp:84:7
```

Differential Revision: D19669832

fbshipit-source-id: 7fb60a06dbb6a04edddb6b619af7a8cef4995fd2
dotconnor pushed a commit to 5448C8/folly that referenced this pull request Mar 19, 2021
@marksantaniello @dotconnor
Use sized deallocation in SysAllocator and Arena
9f487c0 
Summary:
Use sized-deallocation (`sdallocx`) if possible in `folly::SysAllocator` and `folly::Arena`.

`Arena` has always allocated two types of blocks:
1. Normal (fixed-sized): size is the "goodSize adjusted" `minBlockSize`
2. Large (variable-sized): when facebook#1 is too small

Type facebook#2 makes sized-deallocation tricky -- we need somewhere to remember the allocated sizes.

The old code used a single type `Block` and kept a single list. Here I change to have two types and two lists.  The `LargeBlock` has an additional `allocSize` data member.

This makes the Arena object itself 16B larger, but seems better than adding a 4B `allocSize` to each and every block, regardless of type.

Note that, prior to this change, it was possible to `merge()` arenas with different `minBlockSize`.  This is no longer possible.

Reviewed By: yfeldblum

Differential Revision: D22189916

fbshipit-source-id: e6fba48eaae0b5cc8456b856b02d2cfc71c03834
dotconnor pushed a commit to 5448C8/folly that referenced this pull request Mar 19, 2021
@marksantaniello @dotconnor
Use sized deallocation in SysAllocator and Arena (2nd try)
16d0299 
Summary:
Use sized-deallocation (`sdallocx`) if possible in `folly::SysAllocator` and `folly::Arena`.

`Arena` has always allocated two types of blocks:
1. Normal (fixed-sized): size is the "goodSize adjusted" `minBlockSize`
2. Large (variable-sized): when facebook#1 is too small

Type facebook#2 makes sized-deallocation tricky -- we need somewhere to remember the allocated sizes.

The old code used a single type `Block` and kept a single list. Here I change to have two types and two lists.  The `LargeBlock` has an additional `allocSize` data member.

This makes the Arena object itself 16B larger, but seems better than adding a 4B `allocSize` to each and every block, regardless of type.

Note that, prior to this change, it was possible to `merge()` arenas with different `minBlockSize`.  This is no longer possible.

Reviewed By: davidtgoldblatt

Differential Revision: D22466906

fbshipit-source-id: 719f357a0a1f6cfcda3208391837195c3859ab69
@neogenie neogenie mentioned this pull request Jul 28, 2021
Open
facebook-github-bot pushed a commit that referenced this pull request May 10, 2022
@yfeldblum @facebook-github-bot
fix for TSAN-reported data-race in coro::timed_wait
0677a1d 
Summary:
TSAN aborts on a data-race of the form:
```
WARNING: ThreadSanitizer: data race (pid=978596)
  Write of size 8 at 0x7b0400000b40 by thread T2:
    #0 operator delete(void*, unsigned long) <null>
    #1 folly::coro::timed_wait</*...*/>(/*...*/)::'lambda'(/*...*/)::operator()</*...*/>(/*...*/) const folly/experimental/coro/TimedWait.h:51
  Previous atomic write of size 1 at 0x7b0400000b40 by thread T1:
    #0 __tsan_atomic8_exchange <null>
    #1 std::atomic<bool>::exchange(bool, std::memory_order) libgcc/include/c++/bits/atomic_base.h:499
    #2 folly::coro::timed_wait</*...*/>(/*...*/)::'lambda'(/*...*/)::operator()</*...*/>(/*...*/) const folly/experimental/coro/TimedWait.h:64
```

Reviewed By: davidtgoldblatt, iahs

Differential Revision: D36159450

fbshipit-source-id: 860fcbbc1f689f9e166934f75a2ef60abdcad421
markwkern added a commit to markwkern/folly that referenced this pull request Aug 29, 2022
@markwkern
(bench facebook#2 on F4s_v2)
37de19c
@zhngs zhngs mentioned this pull request Nov 21, 2022
Open
facebook-github-bot pushed a commit that referenced this pull request Mar 8, 2023
@giedrbac @facebook-github-bot
Zero size array causes crash in AsyncUDPSocket on llvm15
a5fd234 
Summary:
AsyncUDPSocket test cases are crashing when running under llvm15. During
debugging it seems that the issue is the fact that the code tries to allocated 0
size array. Changing the code to prevent such allocation.

This is not very clean why to fix, but I am not sure if there is better one.
Please let me know if you have any suggestions.

Sample crash:
```
$ buck test //folly/io/async/test:async_udp_socket_test
...
stdout:
Note: Google Test filter = AsyncUDPSocketTest.TestDetachAttach
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from AsyncUDPSocketTest
[ RUN      ] AsyncUDPSocketTest.TestDetachAttach

stderr:
fbcode/folly/io/async/AsyncUDPSocket.cpp:699:10: runtime error: variable length array bound evaluates to non-positive value 0
    #0 0x7f4d8ed93704 in folly::AsyncUDPSocket::writev(folly::SocketAddress const&, iovec const*, unsigned long, int) fbcode/folly/io/async/AsyncUDPSocket.cpp:698
    #1 0x7f4d8ed9081f in folly::AsyncUDPSocket::writeGSO(folly::SocketAddress const&, std::unique_ptr<folly::IOBuf, std::default_delete<folly::IOBuf>> const&, int) fbcode/folly/io/async/AsyncUDPSocket.cpp:528
    #2 0x7f4d8ed900b2 in folly::AsyncUDPSocket::write(folly::SocketAddress const&, std::unique_ptr<folly::IOBuf, std::default_delete<folly::IOBuf>> const&) fbcode/folly/io/async/AsyncUDPSocket.cpp:660
    #3 0x350a05 in AsyncUDPSocketTest_TestDetachAttach_Test::TestBody() fbcode/folly/io/async/test/AsyncUDPSocketTest.cpp:914
    #4 0x7f4d90dd1ad5 in testing::Test::Run() /home/engshare/third-party2/googletest/1.11.0/src/googletest/googletest/src/gtest.cc:2682:50
    #5 0x7f4d90dd1ad5 in testing::Test::Run() /home/engshare/third-party2/googletest/1.11.0/src/googletest/googletest/src/gtest.cc:2672:6
    #6 0x7f4d90dd1c64 in testing::TestInfo::Run() /home/engshare/third-party2/googletest/1.11.0/src/googletest/googletest/src/gtest.cc:2861:14
    #7 0x7f4d90dd1c64 in testing::TestInfo::Run() /home/engshare/third-party2/googletest/1.11.0/src/googletest/googletest/src/gtest.cc:2833:6
    #8 0x7f4d90dd2321 in testing::TestSuite::Run() /home/engshare/third-party2/googletest/1.11.0/src/googletest/googletest/src/gtest.cc:3015:31
    #9 0x7f4d90dd2321 in testing::TestSuite::Run() /home/engshare/third-party2/googletest/1.11.0/src/googletest/googletest/src/gtest.cc:2993:6
    #10 0x7f4d90dd2b1e in testing::internal::UnitTestImpl::RunAllTests() /home/engshare/third-party2/googletest/1.11.0/src/googletest/googletest/src/gtest.cc:5855:47
    #11 0x7f4d90dd1d87 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/engshare/third-party2/googletest/1.11.0/src/googletest/googletest/src/gtest.cc:2665:29
    #12 0x7f4d90dd1d87 in testing::UnitTest::Run() /home/engshare/third-party2/googletest/1.11.0/src/googletest/googletest/src/gtest.cc:5438:55
    #13 0x7f4d90d5c990 in RUN_ALL_TESTS() fbcode/third-party-buck/platform010/build/googletest/include/gtest/gtest.h:2490
    #14 0x7f4d90d5c618 in main fbcode/common/gtest/LightMain.cpp:20
    #15 0x7f4d8ea2c656 in __libc_start_call_main /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #16 0x7f4d8ea2c717 in __libc_start_main@GLIBC_2.2.5 /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../csu/libc-start.c:409:3
    #17 0x33ea60 in _start /home/engshare/third-party2/glibc/2.34/src/glibc-2.34/csu/../sysdeps/x86_64/start.S:116
...
```

Reviewed By: russoue, dmm-fb

Differential Revision: D43858875

fbshipit-source-id: 93749bab17027b6dfc0dbc01b6c183e501a5494c
@liujp liujp mentioned this pull request May 24, 2023
Open
facebook-github-bot pushed a commit that referenced this pull request Apr 27, 2024
@skrueger @facebook-github-bot
Fix TSAN data race in ThreadLocalDetailTest.MultiThreadedTest
58e8d1a 
Summary:
ThreadLocalDetailTest.MultiThreadedTest had a data race on std::vector because multiple threads were accessing and modifying the std::vector without a mutex. I update the code to use indexes such that the vector is not updated concurrently.

Previous run failure that this fixes:

```
stderr:
==================
WARNING: ThreadSanitizer: data race (pid=1249005)
  Write of size 8 at 0x7fff195936d0 by main thread:
    #0 std::vector<std::unique_ptr<folly::test::Barrier, std::default_delete<folly::test::Barrier>>, std::allocator<std::unique_ptr<folly::test::Barrier, std::default_delete<folly::test::Barrier>>>>::pop_back() fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/stl_vector.h:1228 (thread_local_detail_test+0x116c28)
    #1 folly::threadlocal_detail::ThreadLocalDetailTest_MultiThreadedTest_Test::TestBody() fbcode/folly/detail/test/ThreadLocalDetailTest.cpp:121 (thread_local_detail_test+0x101aba)
    #2 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) fbsource/src/gtest.cc:2675 (libthird-party_googletest_1.14.0_gtest.so+0xacf3c)
    #3 testing::Test::Run() fbsource/src/gtest.cc:2692 (libthird-party_googletest_1.14.0_gtest.so+0xacb9c)
    #4 testing::TestInfo::Run() fbsource/src/gtest.cc:2841 (libthird-party_googletest_1.14.0_gtest.so+0xafa4a)
    #5 testing::TestSuite::Run() fbsource/src/gtest.cc:3020 (libthird-party_googletest_1.14.0_gtest.so+0xb4303)
    #6 testing::internal::UnitTestImpl::RunAllTests() fbsource/src/gtest.cc:5925 (libthird-party_googletest_1.14.0_gtest.so+0xd105e)
    #7 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) fbsource/src/gtest.cc:2675 (libthird-party_googletest_1.14.0_gtest.so+0xd08f1)
    #8 testing::UnitTest::Run() fbsource/src/gtest.cc:5489 (libthird-party_googletest_1.14.0_gtest.so+0xd037e)
    #9 RUN_ALL_TESTS() fbsource/gtest/gtest.h:2317 (libcommon_gtest_light_main.so+0x22a7)
    #10 main fbcode/common/gtest/LightMain.cpp:20 (libcommon_gtest_light_main.so+0x2131)

  Previous read of size 8 at 0x7fff195936d0 by thread T123:
    #0 std::vector<std::unique_ptr<folly::test::Barrier, std::default_delete<folly::test::Barrier>>, std::allocator<std::unique_ptr<folly::test::Barrier, std::default_delete<folly::test::Barrier>>>>::size() const fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/stl_vector.h:919 (thread_local_detail_test+0x11b0bd)
    #1 std::vector<std::unique_ptr<folly::test::Barrier, std::default_delete<folly::test::Barrier>>, std::allocator<std::unique_ptr<folly::test::Barrier, std::default_delete<folly::test::Barrier>>>>::operator[](unsigned long) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/stl_vector.h:1045 (thread_local_detail_test+0x11d101)
    #2 folly::threadlocal_detail::ThreadLocalDetailTest_MultiThreadedTest_Test::TestBody()::$_0::operator()() const fbcode/folly/detail/test/ThreadLocalDetailTest.cpp:97 (thread_local_detail_test+0x11d08c)
    #3 void std::__invoke_impl<void, folly::threadlocal_detail::ThreadLocalDetailTest_MultiThreadedTest_Test::TestBody()::$_0>(std::__invoke_other, folly::threadlocal_detail::ThreadLocalDetailTest_MultiThreadedTest_Test::TestBody()::$_0&&) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/invoke.h:61 (thread_local_detail_test+0x11cf95)
    #4 std::__invoke_result<folly::threadlocal_detail::ThreadLocalDetailTest_MultiThreadedTest_Test::TestBody()::$_0>::type std::__invoke<folly::threadlocal_detail::ThreadLocalDetailTest_MultiThreadedTest_Test::TestBody()::$_0>(folly::threadlocal_detail::ThreadLocalDetailTest_MultiThreadedTest_Test::TestBody()::$_0&&) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/invoke.h:96 (thread_local_detail_test+0x11cf05)
    #5 void std::thread::_Invoker<std::tuple<folly::threadlocal_detail::ThreadLocalDetailTest_MultiThreadedTest_Test::TestBody()::$_0>>::_M_invoke<0ul>(std::_Index_tuple<0ul>) fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/std_thread.h:253 (thread_local_detail_test+0x11cebd)
    #6 std::thread::_Invoker<std::tuple<folly::threadlocal_detail::ThreadLocalDetailTest_MultiThreadedTest_Test::TestBody()::$_0>>::operator()() fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/std_thread.h:260 (thread_local_detail_test+0x11ce65)
    #7 std::thread::_State_impl<std::thread::_Invoker<std::tuple<folly::threadlocal_detail::ThreadLocalDetailTest_MultiThreadedTest_Test::TestBody()::$_0>>>::_M_run() fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/std_thread.h:211 (thread_local_detail_test+0x11cd79)
    #8 execute_native_thread_routine /home/engshare/third-party2/libgcc/11.x/src/gcc-11.x/x86_64-facebook-linux/libstdc++-v3/src/c++11/../../../.././libstdc++-v3/src/c++11/thread.cc:82:18 (libstdc++.so.6+0xdf4e4) (BuildId: 452d1cdae868baeeb2fdf1ab140f1c219bf50c6e)

  Location is stack of main thread.

  Location is global '??' at 0x7fff19575000 ([stack]+0x1e6d0)

  Thread T123 (tid=1249233, running) created by main thread at:
    #0 pthread_create <null> (thread_local_detail_test+0x156bef)
    #1 __gthread_create /home/engshare/third-party2/libgcc/11.x/src/gcc-11.x/x86_64-facebook-linux/libstdc++-v3/include/x86_64-facebook-linux/bits/gthr-default.h:663:35 (libstdc++.so.6+0xdf80e) (BuildId: 452d1cdae868baeeb2fdf1ab140f1c219bf50c6e)
    #2 std::thread::_M_start_thread(std::unique_ptr<std::thread::_State, std::default_delete<std::thread::_State>>, void (*)()) /home/engshare/third-party2/libgcc/11.x/src/gcc-11.x/x86_64-facebook-linux/libstdc++-v3/src/c++11/../../../.././libstdc++-v3/src/c++11/thread.cc:147:37 (libstdc++.so.6+0xdf80e)
    #3 folly::threadlocal_detail::ThreadLocalDetailTest_MultiThreadedTest_Test::TestBody() fbcode/folly/detail/test/ThreadLocalDetailTest.cpp:93 (thread_local_detail_test+0x1015cd)
    #4 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) fbsource/src/gtest.cc:2675 (libthird-party_googletest_1.14.0_gtest.so+0xacf3c)
    #5 testing::Test::Run() fbsource/src/gtest.cc:2692 (libthird-party_googletest_1.14.0_gtest.so+0xacb9c)
    #6 testing::TestInfo::Run() fbsource/src/gtest.cc:2841 (libthird-party_googletest_1.14.0_gtest.so+0xafa4a)
    #7 testing::TestSuite::Run() fbsource/src/gtest.cc:3020 (libthird-party_googletest_1.14.0_gtest.so+0xb4303)
    #8 testing::internal::UnitTestImpl::RunAllTests() fbsource/src/gtest.cc:5925 (libthird-party_googletest_1.14.0_gtest.so+0xd105e)
    #9 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) fbsource/src/gtest.cc:2675 (libthird-party_googletest_1.14.0_gtest.so+0xd08f1)
    #10 testing::UnitTest::Run() fbsource/src/gtest.cc:5489 (libthird-party_googletest_1.14.0_gtest.so+0xd037e)
    #11 RUN_ALL_TESTS() fbsource/gtest/gtest.h:2317 (libcommon_gtest_light_main.so+0x22a7)
    #12 main fbcode/common/gtest/LightMain.cpp:20 (libcommon_gtest_light_main.so+0x2131)

ThreadSanitizer: data race fbcode/third-party-buck/platform010/build/libgcc/include/c++/trunk/bits/stl_vector.h:1228 in std::vector<std::unique_ptr<folly::test::Barrier, std::default_delete<folly::test::Barrier>>, std::allocator<std::unique_ptr<folly::test::Barrier, std::default_delete<folly::test::Barrier>>>>::pop_back()
==================
ThreadSanitizer: reported 1 warnings
```

Reviewed By: yfeldblum

Differential Revision: D56643303

fbshipit-source-id: cc364817ae79bc6418cc07faa35e1bcd21830c66
DenisYaroshevskiy added a commit to DenisYaroshevskiy/folly that referenced this pull request Aug 6, 2024
@DenisYaroshevskiy @facebook-github-bot
migrate simdForEach facebook#2: remove unroll step
bd5fd29 
Summary: detail::UnrollStep should go away, because otherwise it would be exposed to the user of `simdForEach`.

Reviewed By: yfeldblum

Differential Revision: D60587842
facebook-github-bot pushed a commit that referenced this pull request Aug 6, 2024
@DenisYaroshevskiy @facebook-github-bot
migrate simdForEach #2: remove unroll step
29a2e9b 
Summary: detail::UnrollStep should go away, because otherwise it would be exposed to the user of `simdForEach`.

Reviewed By: yfeldblum

Differential Revision: D60587842

fbshipit-source-id: 34c75dcbb6ddeefeeae55c98f0656f7008d57e0e
facebook-github-bot pushed a commit that referenced this pull request Aug 12, 2024
@spikeh @facebook-github-bot
folly: io_uring: only call readCallback if it did not change
7ac9203 
Summary:
Ucache with io_uring is crashing with this stack trace:

```
(lldb) bt
* thread #1, name = 'ucache', stop reason = signal SIGSEGV: address not mapped to object
  * frame #0: 0x0000000000000000
    frame #1: 0x00007f517ba44560 libc.so.6`__restore_rt at libc_sigaction.c:13
    frame #2: 0x000000000a16e249 ucache`folly::AsyncIoUringSocket::ReadSqe::callback(this=0x00007f498c5955e0, cqe=<unavailable>) at AsyncIoUringSocket.cpp:639
    frame #3: 0x000000000964196a ucache`folly::IoUringBackend::getActiveEvents(folly::IoUringBackend::WaitForEventsMode) [inlined] folly::IoSqeBase::internalCallback(this=<unavailable>, cqe=<unavailable>) at IoUringBackend.cpp:0
    frame #4: 0x000000000964195c ucache`folly::IoUringBackend::getActiveEvents(folly::IoUringBackend::WaitForEventsMode) [inlined] folly::IoUringBackend::internalProcessCqe(this=0x00007f4a8f989c00, maxGet=4294967295, mode=NORMAL) at IoUringBackend.cpp:1556
    frame #5: 0x000000000964190f ucache`folly::IoUringBackend::getActiveEvents(this=0x00007f4a8f989c00, waitForEvents=<unavailable>) at IoUringBackend.cpp:1527
    frame #6: 0x0000000009640ac9 ucache`folly::IoUringBackend::eb_event_base_loop(this=0x00007f4a8f989c00, flags=1) at IoUringBackend.cpp:1178
    frame #7: 0x000000001b61547c ucache`folly::EventBase::loopMain(this=0x00007f4a8f97af00, flags=0, options=<unavailable>) at EventBase.cpp:0
    frame #8: 0x000000001bab9a2d ucache`folly::EventBase::loopBody(this=0x00007f4a8f97af00, flags=0, options=(ignoreKeepAlive = false, allowSuspension = false)) at EventBase.cpp:513
    frame #9: 0x000000000a3a6c90 ucache`folly::EventBase::loop(this=<unavailable>) at EventBase.cpp:474
    frame #10: 0x000000000a3a75f7 ucache`folly::EventBase::loopForever(this=<unavailable>) at EventBase.cpp:781
    frame #11: 0x000000000a436907 ucache`folly::IOThreadPoolExecutor::threadRun(this=0x00007f4da3fce1c0, thread=<unavailable>) at IOThreadPoolExecutor.cpp:240
    frame #12: 0x0000000009ed68b8 ucache`void std::__invoke_impl<void, void (folly::ThreadPoolExecutor::*&)(std::shared_ptr<folly::ThreadPoolExecutor::Thread>), folly::ThreadPoolExecutor*&, std::shared_ptr<folly::ThreadPoolExecutor::Thread>&>((null)=<unavailable>, __f=<unavailable>, __t=<unavailable>, __args=<unavailable>) at invoke.h:74
    frame #13: 0x00007f517b6df4e5 libstdc++.so.6`std::execute_native_thread_routine(__p=0x00007f4da01bdbe0) at thread.cc:82:18
    frame #14: 0x00007f517ba9abc9 libc.so.6`start_thread(arg=<unavailable>) at pthread_create.c:434:8
    frame #15: 0x00007f517bb2cf5c libc.so.6`__clone3 at clone3.S:81
```

In frame 2 the offending function is `ReadSqe::callback()`: https://fburl.com/code/4fb46hdq

The problem is that `readCallback_` is null:

```
(lldb) fr v readCallback_
(folly::AsyncReader::ReadCallback *) readCallback_ = nullptr
```

However, `readCallback_` is checked earlier in the the function `ReadSqe::callback()`: https://fburl.com/code/dhnndk29

`AsyncIoUringSocket::readError()` now fails all pending writes and calls `writeErr()`, whose implementation can then close out a socket and call `AsyncIoUringSocket::setReadCB(nullptr)` to change the callback.

Move the call to `AsyncIoUringSocket::readError()` after the call to `readCallback_`.

Reviewed By: dmm-fb

Differential Revision: D60880377

fbshipit-source-id: 42d2f430cd3dc3f4787d0e4c29da7c8ef31ba9a8
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@chipturner @jdelong