Amass go API free resources

[g0rbe]

I started to use the Amass API as part of my go code. My code never stops, because it is running the builtin HTTP server.
My code came from the example code.
Amass seems like dont free the allocated sources, ~100 goroutines always added when i call my function and the memory slowly get full.

My function that always been called:

func FindServers(domain string) ([]string, error) {

	rand.Seed(time.Now().UTC().UnixNano())

	cfg := config.NewConfig()
	cfg.AddDomain(domain)

	sys, err := systems.NewLocalSystem(cfg)
	if err != nil {
		return nil, fmt.Errorf("failed to create new local system: %s", err)
	}
	defer sys.Shutdown()

	sys.SetDataSources(datasrcs.GetAllSources(sys, true))

	e := enum.NewEnumeration(cfg, sys)
	if e == nil {
		return nil, fmt.Errorf("failed to create new enumeration")
	}
	defer e.Close()

	err = e.Start()
	if err != nil {
		return nil, fmt.Errorf("failed to start enumeration: %s", err)
	}

	hosts := make([]string, 0)

	for _, o := range e.ExtractOutput(nil, false) {
		if e.Config.IsDomainInScope(o.Name) {
			hosts = append(hosts, o.Name)
		}
	}

	return hosts, nil
}

How to free the memory, stop Amass goroutines after this function return?

[steers]

It seems as though LocalSystem.Shutdown doesn't actually stop the Resolvers it creates, so that may need to be an additional cleanup step. Each baseline resolver spawns several goroutines, so it adds up, especially in this use-case.

Not sure if this was an intentional omission from the Shutdown function, but I'm currently running into this as well.

	sys, err := systems.NewLocalSystem(cfg)
	if err != nil {
		return nil, fmt.Errorf("failed to create new local system: %s", err)
	}
	defer sys.Pool().Stop() // this is the missing piece
	defer sys.Shutdown()

I do see that the call to stop the pool was part of the Shutdown function, but for some reason it seems to be commented out as of v3.11.5, but according to git blame it's been at least 5 months https://github.com/OWASP/Amass/blob/v3.11.5/systems/local.go#L167

It still seems like there's a memory leak somewhere else though, when using Amass as a library like this

[steers]

Looks like the EventBus created as part of the Enumeration doesn't get explicitly cleaned up either, but it's exported so you can call Stop on it directly.

e.Bus.Stop()

These measures alone appear to be ineffective in addressing the memory and/or goroutine leaks though...

[steers]

After doing some profiling (pprof is amazing), it looks like a LOT of the space in use after supposedly everything has been cleaned up is taken by the graph database created by default when instantiating a LocalSystem. Close second seems to be the ASNCache, also in the LocalSystem.

Type: inuse_space
Time: Feb 24, 2021 at 3:07pm (EST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 282.30MB, 97.05% of 290.89MB total
Dropped 52 nodes (cum <= 1.45MB)
Showing top 10 nodes out of 56
      flat  flat%   sum%        cum   cum%
  148.67MB 51.11% 51.11%   148.67MB 51.11%  github.com/tylertreat/BoomFilters.NewBuckets
      48MB 16.50% 67.61%       48MB 16.50%  github.com/AndreasBriese/bbloom.(*Bloom).Size
   25.22MB  8.67% 76.28%    25.22MB  8.67%  github.com/caffix/stringset.Set.Insert (inline)
   24.50MB  8.42% 84.70%    24.50MB  8.42%  bytes.makeSlice
      12MB  4.13% 88.83%    60.13MB 20.67%  github.com/OWASP/Amass/v3/systems.(*LocalSystem).loadCacheData
      10MB  3.44% 92.27%       10MB  3.44%  net.(*IPNet).String
    4.50MB  1.55% 93.81%     4.50MB  1.55%  encoding/csv.(*Reader).readRecord
       4MB  1.38% 95.19%       14MB  4.81%  github.com/caffix/stringset.New
       3MB  1.03% 96.22%        6MB  2.06%  net.ParseCIDR
    2.41MB  0.83% 97.05%    31.63MB 10.87%  github.com/OWASP/Amass/v3/net.(*ASNCache).Update

By default config.NewConfig() sets LocalDatabase to true. For this use case, it's completely irrelevant unless you're migrating data from the enum's own graph database to the local database for persistence (which it doesn't appear that you are).

Ultimately, many of these factors probably stem from this library's primary use in a command-line utility, which typically have little concern for memory stability.

[caffix]

@steers We do migrate the graph database to a local db on disk. It would great to discuss these topics with you further on Discord. Will you join the Amass Discord server?

[caffix]

Is there something left to complete for this issue?

[g0rbe]

No. Sorry for the late.
Thank you @steers