Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LibuvEventEngine Timer Implementation #27513

Closed
wants to merge 232 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
232 commits
Select commit Hold shift + click to select a range
819b500
EventEngine Test Suite: Timers
drfloob Sep 27, 2021
c870887
Rename NoopEventEngine to SimpleEventEngine, which is more the case
drfloob Sep 27, 2021
2c40e94
fix typo
drfloob Sep 27, 2021
9e9cebd
typo
drfloob Sep 27, 2021
ecb29b9
avoid backward reference issues with dependency injection
drfloob Sep 28, 2021
e116331
[WIP] working SimpleEventEngine. Messy. Saved state
drfloob Sep 29, 2021
c6fda02
early thread exit on cancellation; simpler cancel decision logic
drfloob Sep 29, 2021
6cfe237
clean up unused arg warning
drfloob Sep 29, 2021
d761ef3
Squashed commit of the following:
drfloob Sep 29, 2021
1bf9049
EventEngine Test Suite: Timers
drfloob Sep 27, 2021
6e66b23
Rename NoopEventEngine to SimpleEventEngine, which is more the case
drfloob Sep 27, 2021
5f24f72
fix typo
drfloob Sep 27, 2021
5fb231a
typo
drfloob Sep 27, 2021
1258323
LibuvEventEngine Timer Implementation (untested draft)
drfloob Sep 28, 2021
68622cd
Add definitions for unimplemented overriden methods
drfloob Sep 28, 2021
ee81567
Fix thread race on shutdown.
drfloob Sep 28, 2021
1bb897b
avoid backward reference issues with dependency injection
drfloob Sep 28, 2021
8c63696
[WIP] working SimpleEventEngine. Messy. Saved state
drfloob Sep 29, 2021
68734ed
early thread exit on cancellation; simpler cancel decision logic
drfloob Sep 29, 2021
9118b6c
clean up unused arg warning
drfloob Sep 29, 2021
683d2fc
Remove LibuvEventEngine artifacts that aren't related to timers
drfloob Sep 29, 2021
b7b593f
Build on #27496: timer test suite
drfloob Sep 29, 2021
ab32a85
sanitize
drfloob Sep 29, 2021
8f20e27
drop hexdump. (it's neat! should be landed separately)
drfloob Sep 29, 2021
3f286d5
Get asan working, fix ODR
drfloob Sep 29, 2021
ccadb7d
Merge branch 'eventengine-tests' into ee-libuv-timers
drfloob Sep 29, 2021
17a1416
Fix double-close of uv handles on shutdown; add grpc_init for logging
drfloob Sep 29, 2021
f1d4ba3
add 100x timer stress test, asserting now >= scheduled time
drfloob Sep 29, 2021
17714b0
Merge branch 'eventengine-tests' into ee-libuv-timers
drfloob Sep 29, 2021
97dc252
typo
drfloob Sep 29, 2021
7d6263f
Fix thread race on task erasure and task key access
drfloob Sep 29, 2021
c23ae12
test suite does not use polling
drfloob Sep 29, 2021
af1a62b
fix failure accounting and time comparisons
drfloob Sep 29, 2021
594de7d
Merge branch 'eventengine-tests' into ee-libuv-timers
drfloob Sep 29, 2021
e619119
rename impl.cc to libuv_event_engine.cc
drfloob Sep 29, 2021
c11f77c
Test timer scheduling in millis
drfloob Sep 29, 2021
85a0f9e
Merge branch 'eventengine-tests' into ee-libuv-timers
drfloob Sep 29, 2021
2d63308
Fix stale cache and time truncation problems, causing early execution
drfloob Sep 29, 2021
d45749c
Trying to build libuv better on Bazel.
nicolasnoble Sep 29, 2021
cc29a88
100x the timer stress test. made it concurrent. breaks the simpleEE
drfloob Sep 29, 2021
79db0d0
use alternate time math for deadline->timeout
drfloob Sep 29, 2021
eabd318
delete unused test main
drfloob Sep 30, 2021
d35970c
Revert "delete unused test main": wrong test
drfloob Sep 30, 2021
b7dfce4
Kick the can down the road a bit regarding cancel status conversions
drfloob Sep 30, 2021
a5690eb
eliminate thread races
drfloob Sep 30, 2021
28bd115
fix redefinition of SetEventEngineFactory
drfloob Sep 30, 2021
0a81b66
Merge branch 'eventengine-tests' into ee-libuv-timers
drfloob Sep 30, 2021
cd4af9e
Nuke the SimpleEventEngine example. TSAN could not handle 10k threads
drfloob Sep 30, 2021
9cbebe3
Merge branch 'eventengine-tests' into ee-libuv-timers
drfloob Sep 30, 2021
c52418b
sanitize
drfloob Sep 30, 2021
c6380a3
attempt to fix ~0.0025% failure rate of early timer fires
drfloob Sep 30, 2021
baf3563
Merge branch 'master' into ee-libuv-timers
drfloob Oct 4, 2021
d3c5570
cmake config for libuv
drfloob Oct 5, 2021
789d517
Add another libuv include dir
drfloob Oct 5, 2021
c23bc75
ass missing header to libuv bazel build on windows
drfloob Oct 5, 2021
6282ae0
*add
drfloob Oct 5, 2021
587e20f
I accidentally a comma
drfloob Oct 5, 2021
ef1ef08
Support system package libuv; fix cmake linker error
drfloob Oct 5, 2021
26c9128
Merge branch 'master' into ee-libuv-timers
drfloob Oct 5, 2021
72a7ce8
fix libuv build and letsencrypt cert expiry for run_tests_matrix.py
drfloob Oct 5, 2021
ffa6ab7
fix dockerized test build (cmake challenges!)
drfloob Oct 7, 2021
931f9f5
Merge branch 'master' into ee-libuv-timers
drfloob Oct 7, 2021
6930cf5
s/SliceAllocator/MemoryAllocator/g
drfloob Oct 7, 2021
37b3bed
Export uv in cmake
drfloob Oct 7, 2021
4f907c9
tidy
drfloob Oct 8, 2021
b228477
Some python libuv install fixes (untested as of yet)
drfloob Oct 8, 2021
35f6bfc
more cmake tweaks. Need to implement a complicated gen_build_yaml.py
drfloob Oct 8, 2021
1c56450
Attempt at autogeneration of cross-platform libuv wrapper files
drfloob Oct 8, 2021
d104ada
test
drfloob Oct 8, 2021
3233076
Partial application of go/grpc-adding-new-dependency
drfloob Oct 11, 2021
036f7c8
update helloworld cmake example
drfloob Oct 11, 2021
8f09f32
Merge branch 'master' into ee-libuv-timers
drfloob Oct 11, 2021
7f559fb
PHP test fix w/ -std=gnu99 on linux
drfloob Oct 11, 2021
81afc4c
test removal of libuv CMakeLists patch
drfloob Oct 12, 2021
14577a2
attempt to solve darwin build problems
drfloob Oct 12, 2021
5cab339
attempt to fix windows build issues
drfloob Oct 12, 2021
7f83165
yapf & isort
drfloob Oct 12, 2021
e4fb5d6
Add third_party/libuv *.[c,h] to includes
drfloob Oct 12, 2021
a0a6714
remove <thread> dependency
drfloob Oct 12, 2021
59c0fd7
Merge branch 'master' into ee-libuv-timers
drfloob Oct 12, 2021
1b29f54
fix libuv dep hash
drfloob Oct 12, 2021
e8066d1
Merge branch 'master' into ee-libuv-timers
drfloob Oct 12, 2021
e131fa3
fix race on EventEngine destruction
drfloob Oct 13, 2021
df042dc
Join UV on EventEngine destruction
drfloob Oct 13, 2021
7c82be9
python msvc fixes (?)
drfloob Oct 13, 2021
0023501
Add proper PHP includes for libuv
drfloob Oct 13, 2021
7e8c519
Merge branch 'master' into ee-libuv-timers
drfloob Oct 13, 2021
5263865
PHP libs+=psapi; Finduv.cmake logging?
drfloob Oct 13, 2021
a63a7a7
bump to libuv v1.42 to get msys fix (landed in v1.38)
drfloob Oct 14, 2021
1aa7adb
libuv submodule bump
drfloob Oct 14, 2021
7106e42
Run sanitize after upgrading libuv. New files.
drfloob Oct 14, 2021
1c1d234
bump android version (dropping support for KitKat)
drfloob Oct 14, 2021
227e08f
Merge branch 'master' into ee-libuv-timers
drfloob Oct 14, 2021
28e379b
Android version >= 21 everywhere
drfloob Oct 14, 2021
f439b2a
Add more libuv files to our build (incomplete)
drfloob Oct 15, 2021
f1685a0
temporarily remove libuv's sysinfo-loadavg.c (only for MSYS/Cygwin)
drfloob Oct 15, 2021
4ea6d58
libuv headers with headers, sources with sources
drfloob Oct 15, 2021
2746203
Merge branch 'master' into ee-libuv-timers
drfloob Oct 15, 2021
6693eca
[objc] Exclude libuv build source from c-core
dennycd Oct 15, 2021
eaee97f
Merge branch 'master' into ee-libuv-timers
drfloob Oct 18, 2021
ab33d82
attempt to fix mingw build
drfloob Oct 18, 2021
eee61c3
trigger CI to check if 403 resolved
drfloob Oct 18, 2021
3618dd7
add win32 dependencies to setup.py
drfloob Oct 18, 2021
56a787d
fix libuv build file glob (-docs, -test)
drfloob Oct 19, 2021
2cf0bd3
fix mingw linker libs
drfloob Oct 19, 2021
9055163
debug libuv build failure on php
drfloob Oct 20, 2021
d3094ee
remove debug code that breaks some platforms (no `nm`)
drfloob Oct 20, 2021
ffb85dc
debug ruby
drfloob Oct 20, 2021
7df35ff
actually debug ruby
drfloob Oct 20, 2021
6664fe4
Merge branch 'master' into ee-libuv-timers
drfloob Oct 20, 2021
5b21a2c
ruby env wants a string, probably?
drfloob Oct 20, 2021
ff2310e
Merge remote-tracking branch 'dennycd/libuv_ios' into ee-libuv-timers
drfloob Oct 20, 2021
3fc88ad
fix iOS build using Denny's libuv exclusion logic
drfloob Oct 20, 2021
cd2e9a6
more ruby & iOS debug attempts
drfloob Oct 20, 2021
3d8c260
add proctitle back to macOS. Will be problematic for other unixen
drfloob Oct 20, 2021
8568a15
fix proctitle include for mac
drfloob Oct 20, 2021
6fa20ca
change libuv proxy filenames (possibly fixing iOS issues)
drfloob Oct 20, 2021
a532734
generate_projects
drfloob Oct 20, 2021
f83dfa5
py format
drfloob Oct 20, 2021
d4aba1e
attempt iOS fix (remove libuv headers from public set)
drfloob Oct 21, 2021
ece162c
Make Finduv.cmake errors noisier
drfloob Oct 21, 2021
4c0351a
exclude libuv files from gRPC-C++.podspec
drfloob Oct 21, 2021
63bcd38
Fix race around shutdown & kicker_ usage
drfloob Oct 21, 2021
62f11dd
Use Promises instead of Mutexen for consistency
drfloob Oct 21, 2021
76a2695
fix tvos build
drfloob Oct 21, 2021
2f35828
libuv is not project `uv` in cmake =/
drfloob Oct 21, 2021
500183e
target, not library
drfloob Oct 22, 2021
8777c2a
more cmake fixes
drfloob Oct 22, 2021
63de1e4
set libuv dep to Nico's patch (testing)
drfloob Oct 25, 2021
49e6b86
Repomrg -> @yashykt
drfloob Oct 21, 2021
30a5889
Merge branch 'master' into ee-libuv-timers
drfloob Oct 25, 2021
2065360
Merge branch 'master' into ee-libuv-timers
drfloob Oct 26, 2021
f47faf6
libuv with patches from libuv PRs #3340 and #3343
drfloob Oct 27, 2021
cf7a2ea
update libuv submodule commit
drfloob Oct 27, 2021
98a2a8f
cmake external project name uv->libuv
drfloob Oct 27, 2021
2d2147c
fix find_package
drfloob Oct 27, 2021
da529b4
include libuv from non-system paths
drfloob Oct 27, 2021
ac5593b
cmake fix for externalproject helloworld
drfloob Oct 27, 2021
a807b5d
more bazel and cmake fixes
drfloob Oct 28, 2021
ea53ff8
remove DNSubmit
drfloob Oct 28, 2021
22519d2
more cmake fixes ... fixing one build breaks another
drfloob Oct 28, 2021
86b7a6f
CI TESTS ARE GREEN! (small sanity check cleanup)
drfloob Oct 28, 2021
1069952
Making FreeBSD compile again.
nicolasnoble Nov 3, 2021
8c1271b
implement libuv wrapper generator for freebsd
drfloob Nov 3, 2021
808073b
Merge branch 'master' into ee-libuv-timers
drfloob Nov 3, 2021
0a2eb4f
update libuv submodule to HEAD v1.x (includes our patches)
drfloob Nov 3, 2021
e9dac68
sanitize
drfloob Nov 3, 2021
57d39d2
Merge branch 'master' into ee-libuv-timers
drfloob Nov 4, 2021
6c4d47d
add new ConnectionHandle API
drfloob Nov 4, 2021
79caf7e
Clean up documentation; Assert some things only in debug builds.
drfloob Nov 15, 2021
0e26432
Merge branch 'master' into ee-libuv-timers
drfloob Nov 15, 2021
9bcccdd
integrate LibuvEventEngine into the default EE factory
drfloob Nov 15, 2021
50e2b88
reviewer feedback
drfloob Nov 16, 2021
cc133ca
Merge branch 'master' into ee-libuv-timers
drfloob Nov 16, 2021
c0b297e
fix google-global-names-in-headers
drfloob Nov 16, 2021
eaa5985
Merge branch 'master' into ee-libuv-timers
drfloob Nov 18, 2021
317bd00
add missing windows library to linker cmd: user32
drfloob Nov 18, 2021
f16776a
fix for layering_check. GetDefaultEventEngine needs to be exported
drfloob Nov 19, 2021
db20874
Merge branch 'master' into ee-libuv-timers
drfloob Nov 22, 2021
480b340
Merge branch 'master' into ee-libuv-timers
drfloob Nov 30, 2021
c43c552
Merge remote-tracking branch 'origin/master' into ee-libuv-timers
drfloob Feb 3, 2022
f55516a
fix EE deps
drfloob Feb 3, 2022
900b330
Merge branch 'master' into ee-libuv-timers
drfloob Feb 7, 2022
5820d52
Automated change: Fix sanity tests (#121)
drfloob Feb 7, 2022
924439c
grpc::load_reporter
drfloob Feb 16, 2022
b5003f2
Merge branch 'master' into ee-libuv-timers
drfloob Feb 17, 2022
753a73f
Merge remote-tracking branch 'origin/master' into ee-libuv-timers
drfloob Feb 18, 2022
c3f5240
Automated change: Fix sanity tests
drfloob Feb 18, 2022
8f562fe
Merge pull request #139 from drfloob/create-pull-request/patch-753a73…
drfloob Feb 18, 2022
53b2ddc
add libuv to python source distribution
drfloob Feb 19, 2022
46ad20c
Merge remote-tracking branch 'origin/master' into ee-libuv-timers
drfloob Feb 19, 2022
3bad8ea
Merge branch 'master' into ee-libuv-timers
drfloob Feb 23, 2022
0be550c
bring EndpointConfig up to date
drfloob Feb 23, 2022
251972d
revert ares gen_build_yaml (unrelated to PR)
drfloob Feb 23, 2022
b63f5f3
libuv EE: make all overridden methods public as they should be
drfloob Feb 23, 2022
49a1044
move LibuvTask class inside LibuvEventEngine
drfloob Feb 23, 2022
48155bb
make LibuvTask erasing methods private
drfloob Feb 23, 2022
c936bd8
change promise operation names
drfloob Feb 23, 2022
c80cf58
rename libuv EE test suite target
drfloob Feb 23, 2022
afe3aaf
remove task scheduling sanity check (probably not useful)
drfloob Feb 23, 2022
43dbce2
allow the engine to be destroyed from any thread (libuv's included)
drfloob Feb 24, 2022
d005b2d
replace task map with a set of task handles
drfloob Feb 24, 2022
543b989
parens around logical op
drfloob Feb 24, 2022
70b5590
trigger build
drfloob Feb 25, 2022
2cdae2e
Automated change: Fix sanity tests
drfloob Feb 25, 2022
026f8bf
Merge pull request #146 from drfloob/create-pull-request/patch-70b559…
drfloob Feb 25, 2022
f7b5218
remove expensive set.contains check
drfloob Feb 28, 2022
fe862e4
the engine owns the tasks, and can find them via heterogenous lookup
drfloob Mar 1, 2022
4781173
Merge remote-tracking branch 'drfloob/ee-libuv-timers' into ee-libuv-…
drfloob Mar 1, 2022
b230ec1
Allow engines to be destroyed from within the uv thread (callbacks)
drfloob Mar 1, 2022
ccb646b
Merge remote-tracking branch 'origin/master' into ee-libuv-timers
drfloob Mar 1, 2022
bce3224
Automated change: Fix sanity tests
drfloob Mar 1, 2022
801861e
Merge pull request #148 from drfloob/create-pull-request/patch-ccb646…
drfloob Mar 2, 2022
674e025
Merge remote-tracking branch 'origin/master' into ee-libuv-timers
drfloob Mar 2, 2022
cb17c1d
not constexpr
drfloob Mar 2, 2022
10f9a4c
nest LibuvTask::Handle, remove friend; namespace qualification cleanup
drfloob Mar 2, 2022
30a7d52
test fix for cancellation race
drfloob Mar 2, 2022
f676529
comment out unused parameter
drfloob Mar 3, 2022
b894e6f
trigger tests
drfloob Mar 3, 2022
3432adc
Merge remote-tracking branch 'origin/master' into ee-libuv-timers
drfloob Mar 7, 2022
1ea9f95
fix race
drfloob Mar 8, 2022
05af19f
move task set to UvState. Task execution post-EE-destruction is UB
drfloob Mar 8, 2022
2de02f8
purge task queue on shutdown
drfloob Mar 8, 2022
66b71a7
crash on attempts to do work on a defunct engine (RBE debugging)
drfloob Mar 8, 2022
7cd2bc4
RBE fix attempt: avoid accessing destroyed engines when cancelling tasks
drfloob Mar 8, 2022
b71ed49
upgrade third_party/libuv to v1.44.1
drfloob Mar 10, 2022
e4d5e2b
Merge remote-tracking branch 'origin/master' into ee-libuv-timers
drfloob Mar 10, 2022
431188a
Merge branch 'master' into ee-libuv-timers
drfloob Mar 14, 2022
d801149
libuv: android-ifaddrs changes: https://github.com/libuv/libuv/commit…
drfloob Mar 14, 2022
bcb1791
generate projects
drfloob Mar 14, 2022
96a6d49
add EventEngine boot message in production
drfloob Mar 15, 2022
60d7950
Merge branch 'master' into ee-libuv-timers
drfloob Mar 16, 2022
720f713
bump open file limit on fd_conservation test, libuv eats up a few fds
drfloob Mar 16, 2022
55d7ff0
initialize atomic int (thanks valgrind)
drfloob Mar 16, 2022
cfa0c6e
better task logging
drfloob Mar 16, 2022
03e33aa
thread race fix: starting the EventEngine thread reads the time
drfloob Mar 16, 2022
c327c23
Merge branch 'master' into ee-libuv-timers
drfloob Mar 17, 2022
fae32a4
do not trust a handle to be valid
drfloob Mar 17, 2022
5347aee
synchronize around g_now in api_fuzzer
drfloob Mar 17, 2022
25241c4
fix redundant namespace qualifier
drfloob Mar 17, 2022
b3a8e3e
fix ODR
drfloob Mar 17, 2022
bc51a32
DO NOT SUBMIT - attempt libuv cmake fix
drfloob Apr 14, 2022
b3a5828
Merge branch 'master' into ee-libuv-timers
drfloob Apr 14, 2022
fa5a3d6
(ignore the previous DNS - that commit is clean)
drfloob Apr 14, 2022
d0990c8
changed test init constructor signature
drfloob Apr 14, 2022
7f3f759
fix libuv target name
drfloob Apr 14, 2022
c883efc
fix filter_fuzzer data race
drfloob Apr 14, 2022
bfb7a69
fix pkgconfig
drfloob Apr 14, 2022
3dd1493
switch syntax
drfloob Apr 15, 2022
d1c9e29
sanitize
drfloob Apr 15, 2022
4750855
destroy the global default EventEngine on shutdown
drfloob Apr 15, 2022
2af0fa3
safer deletion of the global EventEngine (allows reinitialization)
drfloob Apr 15, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
13 changes: 12 additions & 1 deletion BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -1808,12 +1808,22 @@ grpc_cc_library(
name = "default_event_engine_factory",
srcs = [
"src/core/lib/event_engine/default_event_engine_factory.cc",
"src/core/lib/event_engine/uv/libuv_event_engine.cc",
],
hdrs = [
"src/core/lib/event_engine/uv/libuv_event_engine.h",
"src/core/lib/iomgr/event_engine/promise.h",
],
external_deps = [
# TODO(hork): uv, in a subsequent PR
"absl/container:flat_hash_set",
"absl/strings:str_format",
"libuv",
],
deps = [
"default_event_engine_factory_hdrs",
"event_engine_base_hdrs",
"event_engine_common",
"exec_ctx",
"gpr_base",
],
)
Expand Down Expand Up @@ -2034,6 +2044,7 @@ grpc_cc_library(
"src/core/lib/compression/message_compress.h",
"src/core/lib/debug/stats.h",
"src/core/lib/debug/stats_data.h",
"src/core/lib/event_engine/event_engine_factory.h",
"src/core/lib/event_engine/channel_args_endpoint_config.h",
"src/core/lib/event_engine/sockaddr.h",
"src/core/lib/iomgr/block_annotate.h",
Expand Down
5 changes: 3 additions & 2 deletions BUILDING.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ you will be able to browse and build the code.
> cmake --build . --config Release
```

Using gRPC C++ as a DLL is not recommended, but you can still enable it by running `cmake` with `-DBUILD_SHARED_LIBS=ON`.
Using gRPC C++ as a DLL is not recommended, but you can still enable it by running `cmake` with `-DBUILD_SHARED_LIBS=ON`.

### Windows, Using Ninja (faster build).

Expand Down Expand Up @@ -208,14 +208,15 @@ If you are building gRPC < 1.27 or if you are using CMake < 3.13 you will need
to select "package" mode (rather than "module" mode) for the dependencies.
This means you will need to have external copies of these libraries available
on your system. This [example](test/distrib/cpp/run_distrib_test_cmake.sh) shows
how to install dependencies with cmake before proceeding to installing gRPC itself.
how to install dependencies with cmake before proceeding to installing gRPC itself.

```
# NOTE: all of gRPC's dependencies need to be already installed
$ cmake ../.. -DgRPC_INSTALL=ON \
-DCMAKE_BUILD_TYPE=Release \
-DgRPC_ABSL_PROVIDER=package \
-DgRPC_CARES_PROVIDER=package \
-DgRPC_LIBUV_PROVIDER=package \
-DgRPC_PROTOBUF_PROVIDER=package \
-DgRPC_RE2_PROVIDER=package \
-DgRPC_SSL_PROVIDER=package \
Expand Down