Open-sourcing pushservice

Pushservice is the main recommendation service we use to surface recommendations to our users via notifications. It fetches candidates from various sources, ranks them in order of relevance, and applies filters to determine the best one to send.

README.md

@@ -48,6 +48,16 @@ The core components of the For You Timeline included in this repository are list
 |                          | [visibility-filters](visibilitylib/README.md) | Responsible for filtering Twitter content to support legal compliance, improve product quality, increase user trust, protect revenue through the use of hard-filtering, visible product treatments, and coarse-grained downranking. |
 |                          | [timelineranker](timelineranker/README.md) | Legacy service which provides relevance-scored tweets from the Earlybird Search Index and UTEG service. |
 
+### Recommended Notifications
+
+The core components that power Recommended Notifications included in this repository are listed below:
+
+| Type | Component | Description |
+|------------|------------|------------|
+| Service | [pushservice](pushservice/README.md) | Main recommendation service at Twitter used to surface recommendations to our users via notifications.
+| Ranking | [pushservice-light-ranker](pushservice/src/main/python/models/light_ranking/README.md) | Light Ranker model used by pushservice to rank Tweets. Bridges candidate generation and heavy ranking by pre-selecting highly-relevant candidates from the initial huge candidate pool. |
+|         | [pushservice-heavy-ranker](pushservice/src/main/python/models/heavy_ranking/README.md) | Multi-task learning model to predict the probabilities that the target users will open and engage with the sent notifications. |
+
 ## Build and test code
 
 We include Bazel BUILD files for most components, but not a top-level BUILD or WORKSPACE file. We plan to add a more complete build and test system in the future.

pushservice/BUILD.bazel

@@ -0,0 +1,48 @@
+alias(
+    name = "frigate-pushservice",
+    target = ":frigate-pushservice_lib",
+)
+
+target(
+    name = "frigate-pushservice_lib",
+    dependencies = [
+        "frigate/frigate-pushservice-opensource/src/main/scala/com/twitter/frigate/pushservice",
+    ],
+)
+
+jvm_binary(
+    name = "bin",
+    basename = "frigate-pushservice",
+    main = "com.twitter.frigate.pushservice.PushServiceMain",
+    runtime_platform = "java11",
+    tags = ["bazel-compatible"],
+    dependencies = [
+        "3rdparty/jvm/ch/qos/logback:logback-classic",
+        "finatra/inject/inject-logback/src/main/scala",
+        "frigate/frigate-pushservice-opensource/src/main/scala/com/twitter/frigate/pushservice",
+        "loglens/loglens-logback/src/main/scala/com/twitter/loglens/logback",
+        "twitter-server/logback-classic/src/main/scala",
+    ],
+    excludes = [
+        exclude("com.twitter.translations", "translations-twitter"),
+        exclude("org.apache.hadoop", "hadoop-aws"),
+        exclude("org.tensorflow"),
+        scala_exclude("com.twitter", "ckoia-scala"),
+    ],
+)
+
+jvm_app(
+    name = "bundle",
+    basename = "frigate-pushservice-package-dist",
+    archive = "zip",
+    binary = ":bin",
+    tags = ["bazel-compatible"],
+)
+
+python3_library(
+    name = "mr_model_constants",
+    sources = [
+        "config/deepbird/constants.py",
+    ],
+    tags = ["bazel-compatible"],
+)

pushservice/readme.md

@@ -0,0 +1,45 @@
+# Pushservice
+
+Pushservice is the main push recommendation service at Twitter used to generate recommendation-based notifications for users. It currently powers two functionalities:
+
+- RefreshForPushHandler: This handler determines whether to send a recommendation push to a user based on their ID. It generates the best push recommendation item and coordinates with downstream services to deliver it
+- SendHandler: This handler determines and manage whether send the push to users based on the given target user details and the provided push recommendation item
+
+## Overview
+
+### RefreshForPushHandler
+
+RefreshForPushHandler follows these steps:
+
+- Building Target and checking eligibility
+    - Builds a target user object based on the given user ID
+    - Performs target-level filterings to determine if the target is eligible for a recommendation push
+- Fetch Candidates
+    - Retrieves a list of potential candidates for the push by querying various candidate sources using the target
+- Candidate Hydration
+    - Hydrates the candidate details with batch calls to different downstream services.
+- Pre-rank Filtering, also called Light Filtering
+    - Filters the hydrated candidates with lightweight RPC calls.
+- Rank
+    - Perform feature hydration for candidates and target user
+    - Performs light ranking on candidates
+    - Performs heavy ranking on candidates
+- Take Step, also called Heavy Filtering
+    - Takes the top-ranked candidates one by one and applies heavy filtering until one candidate passes all filter steps
+- Send
+    - Calls the appropriate downstream service to deliver the eligible candidate as a push and in-app notification to the target user
+
+### SendHandler
+
+SendHandler follows these steps:
+
+- Building Target
+    - Builds a target user object based on the given user ID
+- Candidate Hydration
+    - Hydrates the candidate details with batch calls to different downstream services.
+- Feature Hydration
+    - Perform feature hydration for candidates and target user
+- Take Step, also called Heavy Filtering
+    - Perform filterings and validation checking for the given candidate
+- Send
+    - Calls the appropriate downstream service to deliver the given candidate as a push and/or in-app notification to the target user

pushservice/src/main/python/models/heavy_ranking/BUILD

@@ -0,0 +1,169 @@
+python37_binary(
+    name = "update_warm_start_checkpoint",
+    source = "update_warm_start_checkpoint.py",
+    tags = ["no-mypy"],
+    dependencies = [
+        ":deep_norm_lib",
+        "3rdparty/python/_closures/frigate/frigate-pushservice-opensource/src/main/python/models/heavy_ranking:update_warm_start_checkpoint",
+    ],
+)
+
+python3_library(
+    name = "params_lib",
+    sources = ["params.py"],
+    tags = ["no-mypy"],
+    dependencies = [
+        "3rdparty/python/pydantic:default",
+        "src/python/twitter/deepbird/projects/magic_recs/v11/lib:params_lib",
+    ],
+)
+
+python3_library(
+    name = "features_lib",
+    sources = ["features.py"],
+    tags = ["no-mypy"],
+    dependencies = [
+        ":params_lib",
+        "src/python/twitter/deepbird/projects/magic_recs/libs",
+        "twml:twml-nodeps",
+    ],
+)
+
+python3_library(
+    name = "model_pools_lib",
+    sources = ["model_pools.py"],
+    tags = ["no-mypy"],
+    dependencies = [
+        ":features_lib",
+        ":params_lib",
+        "src/python/twitter/deepbird/projects/magic_recs/v11/lib:model_lib",
+    ],
+)
+
+python3_library(
+    name = "graph_lib",
+    sources = ["graph.py"],
+    tags = ["no-mypy"],
+    dependencies = [
+        ":params_lib",
+        "src/python/twitter/deepbird/projects/magic_recs/libs",
+    ],
+)
+
+python3_library(
+    name = "run_args_lib",
+    sources = ["run_args.py"],
+    tags = ["no-mypy"],
+    dependencies = [
+        ":features_lib",
+        ":params_lib",
+        "twml:twml-nodeps",
+    ],
+)
+
+python3_library(
+    name = "deep_norm_lib",
+    sources = ["deep_norm.py"],
+    tags = ["no-mypy"],
+    dependencies = [
+        ":features_lib",
+        ":graph_lib",
+        ":model_pools_lib",
+        ":params_lib",
+        ":run_args_lib",
+        "src/python/twitter/deepbird/projects/magic_recs/libs",
+        "src/python/twitter/deepbird/util/data",
+        "twml:twml-nodeps",
+    ],
+)
+
+python3_library(
+    name = "eval_lib",
+    sources = ["eval.py"],
+    tags = ["no-mypy"],
+    dependencies = [
+        ":features_lib",
+        ":graph_lib",
+        ":model_pools_lib",
+        ":params_lib",
+        ":run_args_lib",
+        "src/python/twitter/deepbird/projects/magic_recs/libs",
+        "twml:twml-nodeps",
+    ],
+)
+
+python37_binary(
+    name = "deep_norm",
+    source = "deep_norm.py",
+    dependencies = [
+        ":deep_norm_lib",
+        "3rdparty/python/_closures/frigate/frigate-pushservice-opensource/src/main/python/models/heavy_ranking:deep_norm",
+        "twml",
+    ],
+)
+
+python37_binary(
+    name = "eval",
+    source = "eval.py",
+    dependencies = [
+        ":eval_lib",
+        "3rdparty/python/_closures/frigate/frigate-pushservice-opensource/src/main/python/models/heavy_ranking:eval",
+        "twml",
+    ],
+)
+
+python3_library(
+    name = "mlwf_libs",
+    tags = ["no-mypy"],
+    dependencies = [
+        ":deep_norm_lib",
+        "twml",
+    ],
+)
+
+python37_binary(
+    name = "train_model",
+    source = "deep_norm.py",
+    dependencies = [
+        ":deep_norm_lib",
+        "3rdparty/python/_closures/frigate/frigate-pushservice-opensource/src/main/python/models/heavy_ranking:train_model",
+    ],
+)
+
+python37_binary(
+    name = "train_model_local",
+    source = "deep_norm.py",
+    dependencies = [
+        ":deep_norm_lib",
+        "3rdparty/python/_closures/frigate/frigate-pushservice-opensource/src/main/python/models/heavy_ranking:train_model_local",
+        "twml",
+    ],
+)
+
+python37_binary(
+    name = "eval_model_local",
+    source = "eval.py",
+    dependencies = [
+        ":eval_lib",
+        "3rdparty/python/_closures/frigate/frigate-pushservice-opensource/src/main/python/models/heavy_ranking:eval_model_local",
+        "twml",
+    ],
+)
+
+python37_binary(
+    name = "eval_model",
+    source = "eval.py",
+    dependencies = [
+        ":eval_lib",
+        "3rdparty/python/_closures/frigate/frigate-pushservice-opensource/src/main/python/models/heavy_ranking:eval_model",
+    ],
+)
+
+python37_binary(
+    name = "mlwf_model",
+    source = "deep_norm.py",
+    dependencies = [
+        ":mlwf_libs",
+        "3rdparty/python/_closures/frigate/frigate-pushservice-opensource/src/main/python/models/heavy_ranking:mlwf_model",
+    ],
+)

pushservice/src/main/python/models/heavy_ranking/README.md

@@ -0,0 +1,20 @@
+# Notification Heavy Ranker Model
+
+## Model Context
+There are 4 major components of Twitter notifications recommendation system: 1) candidate generation 2) light ranking 3) heavy ranking & 4) quality control. This notification heavy ranker model is the core ranking model for the personalised notifications recommendation. It's a multi-task learning model to predict the probabilities that the target users will open and engage with the sent notifications. 
+
+
+## Directory Structure
+- BUILD: this file defines python library dependencies
+- deep_norm.py: this file contains how to set up continuous training, model evaluation and model exporting for the notification heavy ranker model
+- eval.py: the main python entry file to set up the overall model evaluation pipeline
+- features.py: this file contains importing feature list and support functions for feature engineering
+- graph.py: this file defines how to build the tensorflow graph with specified model architecture, loss function and training configuration
+- model_pools.py: this file defines the available model types for the heavy ranker
+- params.py: this file defines hyper-parameters used in the notification heavy ranker 
+- run_args.py: this file defines command line parameters to run model training & evaluation
+- update_warm_start_checkpoint.py: this file contains the support to modify checkpoints of the given saved heavy ranker model
+- lib/BUILD: this file defines python library dependencies for tensorflow model architecture
+- lib/layers.py: this file defines different type of convolution layers to be used in the heavy ranker model
+- lib/model.py: this file defines the module containing ClemNet, the heavy ranker model type
+- lib/params.py: this file defines parameters used in the heavy ranker model