Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate with MaxMind out of the box #766

Merged
merged 12 commits into from
Dec 14, 2020
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -81,3 +81,6 @@ sentry/requirements.txt
relay/credentials.json
relay/config.yml
symbolicator/config.yml
geoip/GeoIP.conf
geoip/*.mmdb
geoip/.geoipupdate.lock
10 changes: 10 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ x-sentry-defaults: &sentry_defaults
volumes:
- 'sentry-data:/data'
- './sentry:/etc/sentry'
- './geoip:/geoip:ro'
x-snuba-defaults: &snuba_defaults
<< : *restart_policy
depends_on:
Expand Down Expand Up @@ -126,6 +127,11 @@ services:
# If you have high volume and your search return incomplete results
# You might want to change this to a higher value (and ensure your host has enough memory)
MAX_MEMORY_USAGE_RATIO: 0.3
geoipupdate:
image: 'maxmindinc/geoipupdate:latest'
entrypoint: ['/usr/bin/geoipupdate', '-d', '/sentry', '-f', '/sentry/GeoIP.conf']
chadwhitacre marked this conversation as resolved.
Show resolved Hide resolved
volumes:
- './geoip:/sentry'
snuba-api:
<< : *snuba_defaults
# Kafka consumer responsible for feeding events into Clickhouse
Expand Down Expand Up @@ -233,6 +239,10 @@ services:
read_only: true
source: ./relay
target: /work/.relay
- type: bind
read_only: true
source: ./geoip
target: /geoip
depends_on:
- kafka
- redis
Expand Down
Binary file added geoip/GeoLite2-City.mmdb.empty
Binary file not shown.
4 changes: 4 additions & 0 deletions install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -325,6 +325,10 @@ if [[ ! -f "$RELAY_CREDENTIALS_JSON" ]]; then
echo "Relay credentials written to $RELAY_CREDENTIALS_JSON"
fi


./install/geoip.sh
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know install.sh is getting a bit unwieldy but does splitting geoip part only help much?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. By splitting out the geoip part I am able to test it in isolation.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ehh, I don't like the fact that we are doing this only for one part of the file. Let's do this only if you have a plan for refactoring the remainder of ./install.sh file without bloating the root directory with many .sh files and replicating all common things across them.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't either. :) I think we should refactor install.sh to make it more modular and therefore more testable ... and we should test it! 😁 I don't see this bloating the root directory with many .sh files, you can see here I split the subscript into an install/ directory. Yes we can and will share components between these scripts ... again, one step at a time.



if [[ "$MINIMIZE_DOWNTIME" ]]; then
# Start the whole setup, except nginx and relay.
$dc up -d --remove-orphans $($dc config --services | grep -v -E '^(nginx|relay)$')
Expand Down
39 changes: 39 additions & 0 deletions install/geoip.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
#!/usr/bin/env bash

if [ ! -f 'install.sh' ]; then echo 'Where are you?'; exit 1; fi

dc="docker-compose --no-ansi"
dcr="$dc run --rm"
chadwhitacre marked this conversation as resolved.
Show resolved Hide resolved


install_geoip() {
local mmdb='geoip/GeoLite2-City.mmdb'
local conf='geoip/GeoIP.conf'
local result='Done'

echo "Setting up IP address geolocation ..."
if [[ ! -f "$mmdb" ]]; then
echo -n "Installing (empty) IP address geolocation database ... "
cp "$mmdb.empty" "$mmdb"
echo "done."
else
echo "IP address geolocation database already exists."
fi

if [[ ! -f "$conf" ]]; then
echo "IP address geolocation is not configured for updates."
echo "See https://develop.sentry.dev/self-hosted/geolocation/ for instructions."
BYK marked this conversation as resolved.
Show resolved Hide resolved
result='Error'
else
echo "IP address geolocation is configured for updates."
echo "Updating IP address geolocation database ... "
$dcr geoipupdate
if [ $? -gt 0 ]; then
result='Error'
fi
echo "$result updating IP address geolocation database."
fi
echo "$result setting up IP address geolocation."
}

install_geoip
2 changes: 1 addition & 1 deletion nginx/nginx.conf
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ http {
proxy_set_header Connection '';
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Request-Id $request_id;
proxy_read_timeout 30s;
Expand Down
1 change: 1 addition & 0 deletions relay/config.example.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,4 @@ processing:
- {name: "bootstrap.servers", value: "kafka:9092"}
- {name: "message.max.bytes", value: 50000000} #50MB or bust
redis: redis://redis:6379
geoip_path: "/geoip/GeoLite2-City.mmdb"
chadwhitacre marked this conversation as resolved.
Show resolved Hide resolved
8 changes: 7 additions & 1 deletion sentry/sentry.conf.example.py
Original file line number Diff line number Diff line change
Expand Up @@ -260,9 +260,15 @@ def get_internal_network():
}
)

#######################
# MaxMind Integration #
#######################

GEOIP_PATH_MMDB = '/geoip/GeoLite2-City.mmdb'

#########################
# Bitbucket Integration #
########################
#########################

# BITBUCKET_CONSUMER_KEY = 'YOUR_BITBUCKET_CONSUMER_KEY'
# BITBUCKET_CONSUMER_SECRET = 'YOUR_BITBUCKET_CONSUMER_SECRET'