Google Cloud Spanner Receiver gracefully handles unreadable databases

[spidercensus]

…

Description:
Fixing a bug. Google Cloud Spanner Receiver currently generates an exception and exits if it attempts to read data from a database that doesn't exist. However it's normal for a single receiver to poll multiple databases, so this is not graceful failure. This PR makes a change to gracefully generate an error in case of an unreadable missing database and then continue reading other databases..

Issue: 14624

Changelog:

• DatabaseReader.Read returns a vector of metadata.MetricsDatapoint and a multierr.Errors object instead of the result vector and a single Error.
• DatabaseReader.Read appends any errors encountered to the multierr.Errors object instead of returning (nil, Error).
• DatabaseReader.Read adds a NewPartialScrapeError to multierr.Errors object if errors were encountered while reading and some results were still populated into the result vector
• ProjectReader.Read returns a vector of metadata.MetricsDatapoint and a Errors object instead of the result vector and nil.
• ProjectReader.Read combines all multierr.Errors objects into a single Error before returning them as err.
• ProjectReader.Read returns a NewPartialScrapeError instead of err if errors were encountered while reading and some results were still populated into the result vector.
• googleCloudSpannerReceiver.Scrape returns a vector of metadata.MetricsDatapoint and a Error object instead of just the result vector.
• All methods mentioned above continue processing metrics after an error is encountered.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: spidercensus / name: Jason Pack (f953470, 935f358, 8289c77, 99795f9, 9ecf8bd, d3a7691, a5425c1, 8c0ae22, e78a7b9, 4a42460, 48d5709, a074ce6, 0f37b36, c3f3649, 306d1a6, 80f9272, a1c579d, 8138873, 81bead8, 799733d, 8da11ad, 161467b, 21bab8a, 1333be8, 98daa7c, 68f063c, 95ec7d0, a7b97c4, c17fb99, 497862f, 4e3f828, 843acba, 2857fd5)
• ✅ login: mx-psi / name: Pablo Baeyens (42baa21)

[songy23]

Please add a changelog

[afarbos]

I close my PR in favor of this one.
I believe we would need 2 thing that I covered in mine (#26860):

1. We need to enable this at the 3 level: systems tables metrics, database and project
2. We need to show this error as partial for the scraper to still retrieve the error. See https://github.com/open-telemetry/opentelemetry-collector/blob/1f5ff4b6b7ccd1f5806c5ce3d50aa1f6196349f2/receiver/scraperhelper/scrapercontroller.go#L188-L214

[spidercensus]

I close my PR in favor of this one. I believe we would need 2 thing that I covered in mine (#26860):

1. We need to enable this at the 3 level: systems tables metrics, database and project
2. We need to show this error as partial for the scraper to still retrieve the error. See https://github.com/open-telemetry/opentelemetry-collector/blob/1f5ff4b6b7ccd1f5806c5ce3d50aa1f6196349f2/receiver/scraperhelper/scrapercontroller.go#L188-L214

I merged in the changes from this PR. Thanks @afarbos

[songy23]

go.mod/go.sum deps changes detected, please run "make gotidy" and commit the changes in this PR.

[spidercensus]

go.mod/go.sum deps changes detected, please run "make gotidy" and commit the changes in this PR.
Thanks, I'll review the tests. This is my first time contributing to this repo.

[spidercensus]

% make -C receiver/googlecloudspannerreceiver do-unit-tests-with-cover
...

go test -race -timeout 300s -parallel 4 --tags="" -coverprofile=coverage.txt -covermode=atomic ./...
ok      github.com/open-telemetry/opentelemetry-collector-contrib/receiver/googlecloudspannerreceiver   1.998s  coverage: 96.3% of statements
ok      github.com/open-telemetry/opentelemetry-collector-contrib/receiver/googlecloudspannerreceiver/internal/datasource       1.543s   coverage: 100.0% of statements
ok      github.com/open-telemetry/opentelemetry-collector-contrib/receiver/googlecloudspannerreceiver/internal/filter   2.580s  coverage: 95.2% of statements
ok      github.com/open-telemetry/opentelemetry-collector-contrib/receiver/googlecloudspannerreceiver/internal/filterfactory    2.162s   coverage: 97.3% of statements
ok      github.com/open-telemetry/opentelemetry-collector-contrib/receiver/googlecloudspannerreceiver/internal/metadata 2.009s  coverage: 97.6% of statements
ok      github.com/open-telemetry/opentelemetry-collector-contrib/receiver/googlecloudspannerreceiver/internal/metadataconfig   2.769s   coverage: [no statements]
ok      github.com/open-telemetry/opentelemetry-collector-contrib/receiver/googlecloudspannerreceiver/internal/metadataparser   2.392s   coverage: 100.0% of statements
ok      github.com/open-telemetry/opentelemetry-collector-contrib/receiver/googlecloudspannerreceiver/internal/statsreader      3.001s   coverage: 64.1% of statements
go tool cover -html=coverage.txt -o coverage.html

% make -C receiver/googlecloudspannerreceiver lint                    
running /Users/.../opentelemetry-collector-contrib/.tools/misspell -error
/Users/.../opentelemetry-collector-contrib/.tools/golangci-lint run --allow-parallel-runners --verbose --build-tags integration --timeout=30m --path-prefix googlecloudspannerreceiver
INFO [config_reader] Config search paths: [./ /Users/.../opentelemetry-collector-contrib/receiver/googlecloudspannerreceiver /Users/.../opentelemetry-collector-contrib/receiver /Users/.../opentelemetry-collector-contrib /Users/... /Users /] 
INFO [config_reader] Used config file ../../.golangci.yml 
INFO [lintersdb] Active 24 linters: [decorder depguard errcheck errorlint exhaustive exportloopref gci gocritic gofmt goimports gosec gosimple govet ineffassign misspell predeclared reassign revive staticcheck tenv unconvert unparam unused wastedassign] 
INFO [loader] Using build tags: [integration]     
INFO [loader] Go packages loading at mode 575 (compiled_files|exports_file|imports|name|types_sizes|deps|files) took 331.151167ms 
INFO [runner/filename_unadjuster] Pre-built 0 adjustments in 3.035833ms 
INFO [linters_context/goanalysis] analyzers took 0s with no stages 
INFO [runner] Issues before processing: 83, after processing: 0 
INFO [runner] Processors filtering stat (out/in): skip_dirs: 83/83, exclude: 83/83, exclude-rules: 0/83, skip_files: 83/83, cgo: 83/83, filename_unadjuster: 83/83, path_prettifier: 83/83, autogenerated_exclude: 83/83, identifier_marker: 83/83 
INFO [runner] processing took 4.107706ms with stages: exclude-rules: 1.704667ms, identifier_marker: 906.458µs, path_prettifier: 671.625µs, autogenerated_exclude: 626.542µs, skip_dirs: 190.5µs, cgo: 3.667µs, filename_unadjuster: 1.458µs, nolint: 584ns, max_same_issues: 458ns, fixer: 292ns, skip_files: 209ns, exclude: 208ns, source_code: 208ns, uniq_by_line: 166ns, severity-rules: 166ns, path_shortener: 84ns, max_per_file_from_linter: 83ns, path_prefixer: 83ns, sort_results: 83ns, diff: 83ns, max_from_linter: 82ns 
INFO [runner] linters took 426.765ms with stages: goanalysis_metalinter: 422.37575ms 
INFO File cache stats: 0 entries of total size 0B 
INFO Memory: 9 samples, avg is 46.8MB, max is 89.9MB 
INFO Execution took 772.344083ms     

[spidercensus]

@harishbohara11 Could you please validate that this works on your end as well?

I used the following config to check this:

exporters:
  logging: {
    verbosity: normal
  }
  prometheus:
    endpoint: '0.0.0.0:9090'
extensions:
  health_check: {}
  memory_ballast:
    size_in_percentage: 10
  pprof:
receivers:
  prometheus/collector:
    config:
      scrape_configs:
        - job_name: "collector-metrics"
          static_configs:
            - targets: ["localhost:8888"]
          scrape_interval: 1m
          scrape_timeout: 10s
          metric_relabel_configs:
  otlp:
    protocols:
      grpc:
        endpoint: null
      http:
        endpoint: null
  googlecloudspanner:
    collection_interval: 60s
    top_metrics_query_max_rows: 100
    backfill_enabled: false
    cardinality_total_limit: 100000
    hide_topn_lockstats_rowrangestartkey: false
    projects:
      - project_id: <redacted>
        instances:
          - instance_id: <redacted>
            databases:
              - <redacted>
              - no_database_on_existing_instance
          - instance_id: non_existent_instance
            databases:
              - no_database_on_non_existent_instance
service:
  telemetry:
    metrics:
      address: :8888
    logs:
      level: info

  extensions:
    - health_check
    - memory_ballast
    - pprof
  pipelines:
    # This pipeline gathers metrics about this collector itself and forwards them to datadog.
    metrics/collector-dd:
      receivers:
        - prometheus/collector
      processors:
        - memory_limiter
        - batch
      exporters:
        - prometheus
    # This pipeline gathers metrics from Spanner and forwards them to datadog.
    metrics:
      receivers:
        - googlecloudspanner
        - otlp
      processors:
        - memory_limiter
        - batch
      exporters:
        - logging
        - prometheus
processors:
  batch:
    send_batch_size: 4000
    timeout: 200ms
    send_batch_max_size: 5000

  memory_limiter:
    limit_percentage: 80
    spike_limit_percentage: 10
    check_interval: 1s

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[harishbohara11]

@bogdandrutu Can you please review the PR? As a codeowner of GoogleCloudSpannerReceiver, @KiranmayiB has provided approval.

[mx-psi]

@spidercensus Thanks for your PR and apologies that it is taking so long to merge this. Given we now have a codeowner approval I can do a final review and merge this, but before this I need you to do the following steps so that CI passes:

1. Merge main into this branch and make sure that the versions for go.opentelemetry.io modules in receiver/googlecloudspannerreceiver/go.mod are v0.90.1 or v1.0.0 for all modules
2. Run make gotidy and make genotelcontribcol
3. Remove the receiver/googlecloudspannerreceiver/coverage.html and receiver/googlecloudspannerreceiver/coverage.txt files

Could you do these steps and ping me when finished? Thanks!

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 14 days.

[spidercensus]

This build process really needs to be made platform-independent (Dockerized). There were far too many times that I had to re-run go.mod because my version of go was slightly different than what was expected by the build tests.

@mx-psi Please review.

[mx-psi]

@spidercensus Thanks for following the steps, there was one more merge conflict but it was easy to fix so I fixed it myself :) I am sorry it took so long to merge this, but it finally got through!

There were far too many times that I had to re-run go.mod because my version of go was slightly different than what was expected by the build tests.

If you have time, would you be able to open an issue about this? If not, any further details are helpful. In general, we aim to have compatibility with all supported Go versions (currently Go 1.20 and Go 1.21), but dependency management in Go can sometimes be a bit unwieldy, and e.g. right now the make gotidy target output can differ between Go 1.20 and Go 1.21. We should at the very least document this, but would like to first confirm that this is the issue you faced