How are folks sourcing their observations (if not using mock-observations plugin)

[heaversm]

I am using the mock observations plugin in my pipeline in order to source the requisite data for further pipeline measurement, but if I were not using this, how would I gather data from, say, my website hosted on GCP?

[jawache]

Hi @heaversm,

This is a common questions, there are a couple of hackathon projects that include creating an importer (aws/gcp etc...) so post hackathon I think we'll have a few more options but for now it's just either manually creating them (we have one org that manually creates an entire impact manifest file dynamically using a script that queries their monitoring db) or using the azure importer that currently exists or just mocking them with mock observations.

A useful attribute of mock observations is that it can be a good signal to data providers about how you want the data to be imported from their platforms and also what data you want exposed. In a way it's almost a spec, "build me something that imports data in this format". So we're hoping that after this we can reach out to them with concrete examples and encourage them to create importers that match the structure that's requires for downstream plugins that use that data. Or perhaps people will just create and maintain community versions of the importers.

Post hackathon we're going to have some deep dive retros/reviews with all participants to gather feedback, refine features, prioritize a backlog etc... Questions like "what's the best way to get data into a manifest file?" is top of mind. We've made some guesses that work for the narrow use cases we have right now but I'm excited to get feedback so we can all collaborate on a better solution (or double down on the existing ones)

Hope that helps!