Skip to content
Permalink
Browse files

draft design docs

  • Loading branch information
narqo committed Sep 30, 2019
1 parent b9aa524 commit 4c8e7346db2b532154819d7097580bb8a0f8de72
Showing with 42 additions and 9 deletions.
  1. +34 −0 DESIGN.md
  2. +1 −9 README.md
  3. +3 −0 docs/profefe1-1.svg
  4. +3 −0 docs/profefe1-2.svg
  5. +1 −0 docs/profefe1.drawio
@@ -0,0 +1,34 @@
# Design

This is the working draft of the system design.

## Components

*profefe* consists of the following components

### Collector

(a.k.a. `cmd/profefe`)

A service that receives profiling data in the [format of pprof](https://github.com/google/pprof/tree/6ce02741cba3adff4f8d4de07dc7a2379334cae4/doc) (`pb.gz`). Collector persists the data in a plugable storage.
It also provides an API for querying and retrieving stored profiles.

The only storage implemented currently uses [Badger](https://github.com/dgraph-io/badger).

*Note: it's tempting to split collector into two separate applications (collector and querier). That might inctease service's scalability. But is the subject of future research.*

### Agent

An optional library, that can be integrated into an application instead of `net/http/prof`. The goal of agent is to periodically scrap pprof data from the running application and to send the data to collector.

## Interaction

Below is the schema of profefe's components interaction:

<img src="./docs/profefe1-1.svg" alt="Schema of profefe components interaction"/>

If integrating the agent into the application is impossible or undesirable, but the application already exposes `net/http/pprof` server, the pprof data can be scrapped using an external mechanism, that will send the data to collector using its HTTP API.

In the following example, the agent was replaced with an external script that is running inside a cronjob. It periodically requests profiling data from application's pprof server and sends it to collector:

<img src="./docs/profefe1-2.svg" alt="Schema of replacing agent with a cronjob"/>
@@ -38,15 +38,7 @@ annotating metrics with labels.

## How does it work?

*profefe* consists of:

- Collector — a service that receives profiles from the agent, stores them in the persistent storage, and provides an API for querying profiles.
- Agent — an optional library that can be integrated into your project. Its goal is to scrap pprof data from the running instance periodically and to send the data to collector.

*Note: it's tempting to split collector into two separate parts (collector and querier) to inctease scalability. This
is the subject of future research.*

Collector requires a plugable storage to persist the profiling data. Currently it only supports [Badger](https://github.com/dgraph-io/badger).
See [Design Docs](DESIGN.md).

## Quickstart

@@ -0,0 +1,3 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" version="1.1" width="713px" height="361px" viewBox="-0.5 -0.5 713 361" content="&lt;mxfile host=&quot;www.draw.io&quot; modified=&quot;2019-09-30T19:06:41.892Z&quot; agent=&quot;Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:69.0) Gecko/20100101 Firefox/69.0&quot; version=&quot;12.0.0&quot; etag=&quot;KmC23s92Qh95r3yuDScW&quot; type=&quot;device&quot; pages=&quot;1&quot;&gt;&lt;diagram id=&quot;2uSIS6w2sEnPm_tKGVOz&quot;&gt;7Vtbc5s4FP41foyHu+ExidP2oTvNTDqz7aMMss0WIyrkxM6vXwkECCEuNpDstiUzCTqSkHSun46UhXl/OH3EINn/hQIYLQwtOC3M9cIwdMNx6B9GOecU1zRzwg6HAW9UEZ7CV8iJGqcewwCmtYYEoYiESZ3ooziGPqnRAMbopd5si6L6qAnYwQbhyQdRk/p3GJA9X4WxquifYLjbFyPrjpfXHEDRmK8k3YMAvQgk82Fh3mOESP52ON3DiDGv4Eve70NLbTkxDGMypIORd3gG0ZGvjc+LnIvF7jA6JrwZxASeVCwGm6K51pyCXi6MagREB0jwmTbhHzKMpZ334dpg8uJLxVpjxWl7ka0WHwtwce7Kb1crpi980WoGmP0MoOuPA8jaawvz7mUfEviUAJ/VvlD9prQ9OdDvr3X6ug2j6B5FCGd9za3Nfig9JRj9gEKNkz2sB4qJQM8fSm9luchao5O13kpibWFAAmv1laZgrTEBa61fmrWy1upNrZ2PtfbMrN1uDd9XsTZwNo7tTMNCV9ZO4y2101Gw0InoCHdB+FxjpfPzyNxxpk03aRaMbmkD3UlOVSWbTRTu4lqPCG6J2MTZsb8gSaLQByREcTEknWw2at6gIUvKY1IXWF0wMYqhJEVOKuaUzcS8Y/KiQ0e3nHwIg4ANotSPugax1fNIrE+kALpZl7/VlL9K/FNIfzWvAQUAululATm+CzfbmQxIH+rfzQlY6I42IIsaUGYWkoHgY0zCA6QTSBKMtm9mJD7lO8TTmYk1kZnYmiTlZqixZrITb+ZA4/pQHWg2rm3Z2kx24r6hmRQmKXAMBnQ3wYsIkz3aoRhEDxX1zj/i54ylekPBqg6fEUp4k38gIWeuduBIUJ3p8BSSb7w7e//O3pc2L61PQtX6XBRius5vYkHoxYpVt6xU9MsFWeyK9FaLYDzoFidlGTpiH9YUkQC8g0Tw4U2hYxjR2Ppc/7pKglnXW4zBWWiQoDAmqfDlR0aodMmua5Jni7rQ11o3bEl18uErRSrXMUy39IZ5MocJt/CGbkHZRH4XH+lJBq4A4/ZMLlJv7qFZ9MK/AfPLXWXTn84VkPQBG/Z0DxL2ejjtWP5puUkO8ZITRfYXXPwMNjB6RGmYQXJzvUGEoAPD76ziDvg/dhkP6zsk+igkQZhPbgosgZjqRPa+pgsKkxQ+VqQ7dCRRGFNNKPJVWkWkpZSAOAA4YCpzPmwQW0CuY5MYj6X0aaI0zbmCYzNHkBzT/RURc7IgWbwPCZKyLTWD37Vh1EeH0Ofve4TDVzoSkPW2Yf0t+sq9SF6LcACxVJNShxHGu68Zq26q3V0jSPdE5GJTPTz5d6MtTVt36lGSK9ZlgbwRey1Ngs2ywuYz570qnb0UEZgr9ThtmEBu73VggqpzMRu03aaQLGQbuxA32O9pYdrSYcwsrYzqgKaZ3WiUFgR/yZEmwOSWZfMFRWa0DyFbODeqQGpBKUJ9P1oVDbFM1rNCANJ9CdFZ4REQOrk4oxhaB9IVjUgfbjEjzcE0vaXrGqZteJaurxxHVtplUWOtNNt1BtmKchRTeIz6MIbtLR3d06wV/10fJefMeIu06hZWgJHJYHd77pCpz6Dch1FPHuY5jccvT1+FbEf+sSLdcVluxbgyOTnNQujkQRKyBA7djoQRTHtWtcFjxppmxlN8hSoRCAABPcttJ9cSXP8niTNBL5PNcvd64SJbsHuMSNt+RggpmxLrfCnBcu6P8Y8vDNSQLIIsNVty7Sz+pIKb78mT56iefeADOIQR++gnGD1D1qjCT5RqD0XkPWeirln3YApM7ulNTF7uxEaB8tVodCBGzVFIQfe8Gh7XssB6IVAQw/YQ1JCV+eScASiCloTWakyh1BtZIyfCFO5bYQqnE1N49rKqoqBidR2kcBxj6VnVI50ZumwKwjMPpHBkSNGF2q+BFO2nKeP88seHP4jiD6L4E2w7gq0tJY9VwbYMrGK0LS4ljQq2A07Y8htYfemdVk6Mv6JlmXXvZytuaHnKhK+znIBHxYRnvqXmdfLAlI9yyqUJXHAUTHAngGRG86Cn1Zj9M7XNIMsN9xl0jtk+b/osXDq13WjQhBXUEWo06GquO9QshRuPNY6/BUObhzbrX+tKQQtvix51g1ZcuZzrAEdx4zKAzzBCieLIrNDp4yG69QkSGXnpoQ0XjDrYCBJWHsVcJYIeh1LqdvvdTNWxizOFDAZczZzPoYphVbXCSex7wA3J8SvM+fhOK1QlIaUV/ieu5tjvyKPmNb/ifoSPoghmLuVXcvotvC6Abt3nKy5BGXM5fbfB5rc4cCoPX6vz1u8L8bh1yB0mo1UanXeYCmAvZmYKm62dmY5NzQyWgfcuMrjiWL3/VK7/XE/I8vkRSNPQlxJ9nQrSez6fleREY5UH1DvO0K9SEnOkkqizWjIMlDe0LSm6K/JbhbMRPPHPI+QK3O19MUzDVx6GGZd5qo62tu8W9voiv9uNDWUta78wMQ41dONCfhwuyuWGq8lIccvXYG8KLD7mcgEtVv+uljev/unPfPgX&lt;/diagram&gt;&lt;/mxfile&gt;"><defs/><g><rect x="98" y="10" width="170" height="120" fill="#f5f5f5" stroke="#666666" pointer-events="none"/><rect x="93" y="15" width="170" height="120" fill="#f5f5f5" stroke="#666666" pointer-events="none"/><rect x="88" y="20" width="170" height="120" fill="#fff2cc" stroke="#d6b656" pointer-events="none"/><g transform="translate(137.5,41.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="77" height="17" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 16px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 78px; white-space: nowrap; overflow-wrap: normal; text-align: left;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;white-space:normal;"><div style="font-size: 16px" align="left">application</div></div></div></foreignObject><text x="39" y="17" fill="#000000" text-anchor="middle" font-size="16px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><rect x="88" y="110" width="170" height="30" fill="#dae8fc" stroke="#6c8ebf" pointer-events="none"/><g transform="translate(129.5,117.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="82" height="15" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 14px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 83px; white-space: nowrap; overflow-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;white-space:normal;"><div style="font-size: 14px">runtime/pprof</div></div></div></foreignObject><text x="41" y="15" fill="#000000" text-anchor="middle" font-size="14px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><rect x="88" y="80" width="170" height="30" fill="#f8cecc" stroke="#b85450" pointer-events="none"/><path d="M 88.47 95.53 Q 55.53 95.53 55.53 110.24 Q 55.53 124.94 82.1 124.94" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 87.35 124.94 L 80.35 128.44 L 82.1 124.94 L 80.35 121.44 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g transform="translate(128.5,87.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="83" height="15" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 14px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 84px; white-space: nowrap; overflow-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;white-space:normal;">profefe-agent</div></div></foreignObject><text x="42" y="15" fill="#000000" text-anchor="middle" font-size="14px" font-family="Helvetica">profefe-agent</text></switch></g><g transform="translate(4.5,102.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="31" height="15" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 14px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 32px; white-space: nowrap; overflow-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;white-space:normal;">timer</div></div></foreignObject><text x="16" y="15" fill="#000000" text-anchor="middle" font-size="14px" font-family="Helvetica">timer</text></switch></g><ellipse cx="60.5" cy="110" rx="15" ry="15" fill="#ffffff" stroke="#000000" pointer-events="none"/><ellipse cx="60.5" cy="110" rx="11.700000000000001" ry="11.700000000000001" fill="#ffffff" stroke="#000000" pointer-events="none"/><path d="M 60.5 98.3 L 60.5 99.8 M 66.31 99.83 L 65.47 101.3 M 70.56 104.08 L 69.09 104.98 M 72.2 110 L 70.65 110 M 70.56 115.87 L 69.09 114.97 M 66.31 120.12 L 65.47 118.66 M 60.5 120.15 L 60.5 121.7 M 54.64 120.12 L 55.48 118.66 M 50.39 115.87 L 51.86 114.97 M 48.8 110 L 50.3 110 M 50.39 104.08 L 51.86 104.98 M 54.64 99.83 L 55.48 101.3 M 61.08 99.95 L 60.5 110 L 67.06 110.28" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 257.88 95.53 L 348.47 95.53 L 348.47 60.24 L 402.1 60.24" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 407.35 60.24 L 400.35 63.74 L 402.1 60.24 L 400.35 56.74 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g transform="translate(302.5,74.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="30" height="15" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 14px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; white-space: nowrap; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;">push</div></div></foreignObject><text x="15" y="15" fill="#000000" text-anchor="middle" font-size="14px" font-family="Helvetica">push</text></switch></g><path d="M 317.88 260.24 L 317.88 204.94 L 317.88 95.53" fill="none" stroke="#000000" stroke-miterlimit="10" stroke-dasharray="1 4" pointer-events="none"/><path d="M 261 260 L 338 260 L 352 274 L 352 360 L 261 360 L 261 260 Z" fill="#ffffff" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 338 260 L 338 274 L 352 274 Z" fill-opacity="0.05" fill="#000000" stroke="none" pointer-events="none"/><path d="M 338 260 L 338 274 L 352 274" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g transform="translate(265.5,269.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="61" height="56" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 62px; white-space: nowrap; overflow-wrap: normal; text-align: left;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;white-space:normal;"><font style="font-size: 12px">POST</font><div style="font-size: 12px" align="left"><font style="font-size: 12px">/api/profiles</font><br style="font-size: 12px" /><font style="font-size: 12px"><font style="font-size: 12px">metadata</font></font></div><div style="font-size: 12px" align="left"><font style="font-size: 12px">prof.pb.gz</font></div></div></div></foreignObject><text x="31" y="34" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><path d="M 593.18 260.24 L 617.88 260.24 L 617.88 60.24" fill="none" stroke="#000000" stroke-miterlimit="10" stroke-dasharray="1 4" pointer-events="none"/><path d="M 573 260 L 659 260 L 673 274 L 673 335 L 573 335 L 573 260 Z" fill="#ffffff" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 659 260 L 659 274 L 673 274 Z" fill-opacity="0.05" fill="#000000" stroke="none" pointer-events="none"/><path d="M 659 260 L 659 274 L 673 274" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g transform="translate(577.5,269.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="61" height="41" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 12px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 62px; white-space: nowrap; overflow-wrap: normal; text-align: left;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;white-space:normal;"><font style="font-size: 12px">GET</font><div style="font-size: 12px" align="left"><font style="font-size: 12px">/api/profiles</font><br style="font-size: 12px" /><font style="font-size: 12px"><font style="font-size: 12px">metadata</font></font></div></div></div></foreignObject><text x="31" y="27" fill="#000000" text-anchor="middle" font-size="12px" font-family="Helvetica">[Not supported by viewer]</text></switch></g><path d="M 443 162.5 C 443 141.17 503 141.17 503 162.5 L 503 210.5 C 503 231.83 443 231.83 443 210.5 Z" fill="#b0e3e6" stroke="#0e8088" stroke-miterlimit="10" pointer-events="none"/><path d="M 443 162.5 C 443 178.5 503 178.5 503 162.5" fill="none" stroke="#0e8088" stroke-miterlimit="10" pointer-events="none"/><g transform="translate(463.5,183.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="19" height="15" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 14px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 20px; white-space: nowrap; overflow-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;white-space:normal;">DB</div></div></foreignObject><text x="10" y="15" fill="#000000" text-anchor="middle" font-size="14px" font-family="Helvetica">DB</text></switch></g><ellipse cx="683" cy="37.5" rx="7.5" ry="7.5" fill="#ffffff" stroke="#000000" pointer-events="none"/><path d="M 683 45 L 683 70 M 683 50 L 668 50 M 683 50 L 698 50 M 683 70 L 668 90 M 683 70 L 698 90" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g transform="translate(651.5,96.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="61" height="15" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 14px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; white-space: nowrap; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;">developer</div></div></foreignObject><text x="31" y="15" fill="#000000" text-anchor="middle" font-size="14px" font-family="Helvetica">developer</text></switch></g><rect x="408" y="20" width="130" height="80" fill="#f8cecc" stroke="#b85450" pointer-events="none"/><g transform="translate(422.5,52.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="100" height="15" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 14px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; width: 101px; white-space: nowrap; overflow-wrap: normal; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;white-space:normal;">profefe-collector</div></div></foreignObject><text x="50" y="15" fill="#000000" text-anchor="middle" font-size="14px" font-family="Helvetica">profefe-collector</text></switch></g><path d="M 473.18 100 L 473.18 123.76 L 473.18 137.88" fill="none" stroke="#000000" stroke-width="2" stroke-miterlimit="10" pointer-events="none"/><path d="M 473.18 143.88 L 469.18 135.88 L 473.18 137.88 L 477.18 135.88 Z" fill="#000000" stroke="#000000" stroke-width="2" stroke-miterlimit="10" pointer-events="none"/><path d="M 544.25 60.24 L 682.59 60.24" fill="none" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><path d="M 539 60.24 L 546 56.74 L 544.25 60.24 L 546 63.74 Z" fill="#000000" stroke="#000000" stroke-miterlimit="10" pointer-events="none"/><g transform="translate(599.5,38.5)"><switch><foreignObject style="overflow:visible;" pointer-events="all" width="35" height="15" requiredFeatures="http://www.w3.org/TR/SVG11/feature#Extensibility"><div xmlns="http://www.w3.org/1999/xhtml" style="display: inline-block; font-size: 14px; font-family: Helvetica; color: rgb(0, 0, 0); line-height: 1.2; vertical-align: top; white-space: nowrap; text-align: center;"><div xmlns="http://www.w3.org/1999/xhtml" style="display:inline-block;text-align:inherit;text-decoration:inherit;">query</div></div></foreignObject><text x="18" y="15" fill="#000000" text-anchor="middle" font-size="14px" font-family="Helvetica">query</text></switch></g></g></svg>

0 comments on commit 4c8e734

Please sign in to comment.
You can’t perform that action at this time.