Fix socket leak

[Nick-Triller]

Hi,

this MR fixes the socket leak reported in #52 and #45

Additionally, I refactored some of the code minimally. I took care to create succinct commits.

Thanks
Nick

[dkulchinsky]

Hey @Nick-Triller, just built from your branch and deployed in sandbox, but I started getting these errors:

level=error ts=2020-10-02T23:57:30.586Z caller=harbor_exporter.go:249 msg="Error handling request for /scans/all/metrics" http-statuscode="401 Unauthorized"
level=error ts=2020-10-02T23:57:30.586Z caller=metrics_scan.go:25 unexpectedendofJSONinput=(MISSING)
level=error ts=2020-10-02T23:57:32.140Z caller=harbor_exporter.go:249 msg="Error handling request for /statistics" http-statuscode="401 Unauthorized"
level=error ts=2020-10-02T23:57:32.140Z caller=metrics_statistics.go:27 unexpectedendofJSONinput=(MISSING)
level=error ts=2020-10-02T23:57:33.691Z caller=harbor_exporter.go:249 msg="Error handling request for /systeminfo/volumes" http-statuscode="401 Unauthorized"
level=error ts=2020-10-02T23:57:33.691Z caller=metrics_systemvolumes.go:21 unexpectedendofJSONinput=(MISSING)
level=error ts=2020-10-02T23:57:35.258Z caller=harbor_exporter.go:249 msg="Error handling request for /quotas?page=1&page_size=500" http-statuscode="401 Unauthorized"
level=error ts=2020-10-02T23:57:35.258Z caller=metrics_quotas.go:43 unexpectedendofJSONinput=(MISSING)
level=error ts=2020-10-02T23:57:36.823Z caller=harbor_exporter.go:249 msg="Error handling request for /projects?page=1&page_size=500" http-statuscode="422 Unprocessable Entity"
level=error ts=2020-10-02T23:57:36.823Z caller=metrics_repositories.go:64 unexpectedendofJSONinput=(MISSING)
level=error ts=2020-10-02T23:57:38.373Z caller=harbor_exporter.go:249 msg="Error handling request for /replication/policies?page=1&page_size=500" http-statuscode="401 Unauthorized"
level=error ts=2020-10-02T23:57:38.373Z caller=metrics_replications.go:39 msg="Error retrieving replication policies" err="unexpected end of JSON input"

The exporter is returning some of the metrics, but looks like some collectors are failing.

Any ideas?

[Nick-Triller]

Hi @dkulchinsky, that's interesting, I am not seeing anything like that. Before I start looking into this, could you please test head of master, too?

[dkulchinsky]

Hey @Nick-Triller, thanks for replying! my bad here... sorry for the noise 🤦 had an issue with password injection, so no more 401 errors.

though I'm confused why the exporter was still able to retrieve "some" of the data, I suppose some api endpoints do not require auth? anyway, will monitor memory usage for a while and report back.

---

this seems unrelated but was still getting this error on every scrape:

level=error ts=2020-10-05T13:12:30.165Z caller=harbor_exporter.go:249 msg="Error handling request for /projects?page=1&page_size=500" http-statuscode="422 Unprocessable Entity"
level=error ts=2020-10-05T13:12:30.165Z caller=metrics_repositories.go:64 unexpectedendofJSONinput=(MISSING)

managed to "fix" it by reducing HARBOR_PAGESIZE to 100 (defaults to 500), looking at the api swagger it does seem to suggest a maximum page size of 100 https://github.com/goharbor/harbor/blob/716625a76907aff8c9e5d7924d0c29fee26283d3/api/v2.0/swagger.yaml#L1499-L1507

  pageSize:
    name: page_size
    in: query
    type: integer
    format: int64
    required: false
    description: The size of per page
    default: 10
    maximum: 100

I'll open a separate issue for this.

[dkulchinsky]

@Nick-Triller this has been running for 12 hours now and seems solid to me, great job 👍

@c4po would be awesome if this could be merged 🙏

[c4po]

@Nick-Triller @dkulchinsky thanks for your help. I'll try to get on this tonight or tomorrow.