Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compile in geoip2 database and avoid external file reading #281

Merged
merged 1 commit into from Oct 16, 2018

Conversation

Projects
None yet
6 participants
@tadovas
Copy link
Member

commented Jul 4, 2018

Idea is to reduce external file dependencies and to build-in as much data/info as possible. First candidate geoip database.
Short summary:

  1. Geoip database is converted to few source files by generator.go
  2. New option for builtin country resolving for both client/server - used by default
  3. No need to package db for runtime anymore

@tadovas tadovas requested review from shroomist, donce, Waldz and zolia Jul 4, 2018

@Waldz

This comment has been minimized.

Copy link
Member

commented Jul 5, 2018

+100 kudos, awesome idea

@Waldz

This comment has been minimized.

@tadovas tadovas force-pushed the improvement/build-in-geoip-db branch 6 times, most recently from 68d1164 to 6cf482b Jul 9, 2018

@tadovas tadovas changed the title WIP: Compile in geoip2 database and avoid external file reading Compile in geoip2 database and avoid external file reading Jul 13, 2018

@tadovas

This comment has been minimized.

Copy link
Member Author

commented Jul 13, 2018

@zolia @Waldz this one is ready for review

@Waldz Waldz changed the title Compile in geoip2 database and avoid external file reading WIP Compile in geoip2 database and avoid external file reading Aug 15, 2018

@donce
Copy link
Contributor

left a comment

This PR is shown for me on each pull request reminder, so I'm putting request changes to hide it.

@zolia
Copy link
Member

left a comment

lgtm, just a note that we will need to remove 'bin/common_package/GeoLite2-Country.mmdb' and all related packaging references. Maybe in separate task.

@stale

This comment has been minimized.

Copy link

commented Oct 12, 2018

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Oct 12, 2018

@tadovas tadovas changed the title WIP Compile in geoip2 database and avoid external file reading Compile in geoip2 database and avoid external file reading Oct 15, 2018

@stale stale bot removed the stale label Oct 15, 2018

@tadovas

This comment has been minimized.

Copy link
Member Author

commented Oct 15, 2018

It comes a bit important now

@tadovas tadovas force-pushed the improvement/build-in-geoip-db branch 5 times, most recently from eb26da5 to 4d9a653 Oct 15, 2018

@tadovas

This comment has been minimized.

Copy link
Member Author

commented Oct 15, 2018

@mysteriumnetwork/core you can all take a look, while its being verified by CI

@tadovas

This comment has been minimized.

Copy link
Member Author

commented Oct 15, 2018

@Waldz @vkuznecovas ping. Ready for review

@@ -28,8 +28,12 @@ var (
Usage: "Address (URL form) of ipify service",
Value: "https://api.ipify.org/",
}
locationBuiltInDbFlag = cli.BoolTFlag{

This comment has been minimized.

Copy link
@Waldz

Waldz Oct 15, 2018

Member

Dont need this flag, compiled-in should be default

This comment has been minimized.

Copy link
@tadovas

tadovas Oct 16, 2018

Author Member

Removed

cmd/di.go Outdated
di.LocationResolver = location.NewResolverFake(options.Country)
di.LocationResolver = location.NewStaticResolver(options.Country)
case options.BuiltInDb:
di.LocationResolver = location.NewBuiltInResolver()

This comment has been minimized.

Copy link
@Waldz

Waldz Oct 15, 2018

Member

NewBuiltInResolver should be default resolver if no other option other resolver given

This comment has been minimized.

Copy link
@tadovas

tadovas Oct 16, 2018

Author Member

Done

return "", errors.New("failed to parse IP")
}

countryRecord, err := r.db.Country(ipObject)

This comment has been minimized.

Copy link
@Waldz

Waldz Oct 15, 2018

Member

Looks like duplicated logic, lets DRY with composition:

DBResolver{
  db resource
}

BuiltinDBResolver {
    Resolver DBResolver
}

This comment has been minimized.

Copy link
@tadovas

tadovas Oct 16, 2018

Author Member

DRYied

Country string
IpifyUrl string
ExternalDb string
BuiltInDb bool

This comment has been minimized.

Copy link
@Waldz

Waldz Oct 15, 2018

Member

If build-in is default, we dont need it

This comment has been minimized.

Copy link
@tadovas

tadovas Oct 16, 2018

Author Member

True. Removed


// NewBuiltInResolver returns new BuiltInDbResolver
func NewBuiltInResolver() Resolver {

This comment has been minimized.

Copy link
@vkuznecovas

vkuznecovas Oct 15, 2018

Contributor

the empty line at the start of the func seems odd.

This comment has been minimized.

Copy link
@tadovas

tadovas Oct 16, 2018

Author Member

Removed


// ResolveCountry maps given ip to country
func (r *BuiltInDbResolver) ResolveCountry(ip string) (string, error) {

This comment has been minimized.

Copy link
@vkuznecovas

vkuznecovas Oct 15, 2018

Contributor

the empty line at the start of the func seems odd.

This comment has been minimized.

Copy link
@tadovas

tadovas Oct 16, 2018

Author Member

Removed


country := countryRecord.Country.IsoCode
if country == "" {
country = countryRecord.RegisteredCountry.IsoCode

This comment has been minimized.

Copy link
@vkuznecovas

vkuznecovas Oct 15, 2018

Contributor

if country = countryRecord.RegisteredCountry.IsoCode; country == "" {
return "", errors.New("failed to resolve country")
}

This comment has been minimized.

Copy link
@tadovas

tadovas Oct 15, 2018

Author Member

Well I am not sure if this looks more readable :)

@tadovas tadovas force-pushed the improvement/build-in-geoip-db branch 2 times, most recently from 2b7555b to e4aed78 Oct 15, 2018

@tadovas

This comment has been minimized.

Copy link
Member Author

commented Oct 15, 2018

@Waldz @vkuznecovas all comments addressed. Ready for re-review

@tadovas tadovas force-pushed the improvement/build-in-geoip-db branch from e4aed78 to dde38ea Oct 15, 2018

@soffokl
Copy link
Member

left a comment

btw, did we try to use existing libraries that allows embedding files into executable binary?

cmd/di.go Outdated
@@ -43,6 +41,7 @@ import (
"github.com/mysteriumnetwork/node/services/openvpn"
openvpn_service "github.com/mysteriumnetwork/node/services/openvpn/service"
"github.com/mysteriumnetwork/node/session"
"path/filepath"

This comment has been minimized.

Copy link
@soffokl

soffokl Oct 16, 2018

Member

This does not look goimports formatted.

This comment has been minimized.

Copy link
@soffokl

soffokl Oct 16, 2018

Member

bin/check_goimports: line 23: goimports: command not found
hm, looks like we have broken goimports check.

This comment has been minimized.

Copy link
@soffokl

soffokl Oct 16, 2018

Member

Created issue for fixing goimports on the CI.

This comment has been minimized.

Copy link
@tadovas

tadovas Oct 16, 2018

Author Member

Fixed code.

Show resolved Hide resolved core/location/built_in_resolver_test.go

// EncodedDataLoader returns emmbeded database as byte array
func EncodedDataLoader(data string, originalSize int, compressed bool) (decompressed []byte, err error) {
var reader io.Reader

This comment has been minimized.

Copy link
@soffokl

soffokl Oct 16, 2018

Member

This can be just reader := base64.NewDecoder(base64.RawStdEncoding, strings.NewReader(data))

This comment has been minimized.

Copy link
@tadovas

tadovas Oct 16, 2018

Author Member

It cannot. Because of the compressed flag, reader can be further decorated with gzip.Reader

This comment has been minimized.

Copy link
@tadovas

tadovas Oct 16, 2018

Author Member

Sorry. Got your point. It can be just reader := ...
Thanks.

@tadovas

This comment has been minimized.

Copy link
Member Author

commented Oct 16, 2018

Yea I checked a few embedding libraries, didn't find small and efficient enough, they all were either to big, ugly to use, or not compatible with go generate style integration

@tadovas tadovas force-pushed the improvement/build-in-geoip-db branch from dde38ea to 3206da2 Oct 16, 2018

@soffokl
Copy link
Member

left a comment

Looks good to me.

const originalSize = 3303090
const dbData = "" + dbDataPart0 + dbDataPart1

// LoadData returns emmbeded database as byte array

This comment has been minimized.

Copy link
@soffokl

soffokl Oct 16, 2018

Member

typo: emmbeded -> embedded

This comment has been minimized.

Copy link
@tadovas

tadovas Oct 16, 2018

Author Member

Fixed

@@ -29,9 +29,9 @@ var (
Value: "https://api.ipify.org/",
}
locationDatabaseFlag = cli.StringFlag{
Name: "location.database",
Name: "location.db.external",

This comment has been minimized.

Copy link
@Waldz

Waldz Oct 16, 2018

Member

Lets not change contract

This comment has been minimized.

Copy link
@Waldz

Waldz Oct 16, 2018

Member

location.database

This comment has been minimized.

Copy link
@tadovas

tadovas Oct 16, 2018

Author Member

Fixed

@tadovas tadovas dismissed stale reviews from soffokl and vkuznecovas via 537a4e3 Oct 16, 2018

@tadovas tadovas force-pushed the improvement/build-in-geoip-db branch from 3206da2 to 537a4e3 Oct 16, 2018

@Waldz

Waldz approved these changes Oct 16, 2018

@tadovas

This comment has been minimized.

Copy link
Member Author

commented Oct 16, 2018

@soffokl @vkuznecovas reapprove please

@tadovas tadovas merged commit c61e995 into master Oct 16, 2018

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@tadovas tadovas deleted the improvement/build-in-geoip-db branch Oct 16, 2018

@Waldz Waldz referenced this pull request Oct 24, 2018

Merged

Initial mobile entrypoint for iOS/Swift #447

9 of 13 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.