Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Provide scrapers for Azure CDN, Cache for Redis Enterprise, Database for MariaDB, Load Balancer & Virtual Network #1883

Merged
merged 12 commits into from Dec 17, 2021
Merged

feat: Provide scrapers for Azure CDN, Cache for Redis Enterprise, Database for MariaDB, Load Balancer & Virtual Network #1883

merged 12 commits into from Dec 17, 2021

Conversation

tomkerkhove
Copy link
Owner

@tomkerkhove tomkerkhove commented Dec 14, 2021 •
edited

When implementing a new scraper; these tasks are completed:

  • Implement configuration
  • Implement validation
  • Implement scraping
  • Implement resource discovery
  • Provide unit tests
  • Test end-to-end
  • Document scraper
  • Add entry to changelog

Metrics output:

# HELP promitor_demo_redis_enterprise_memory_declared Amount of requests at the origin of our Azure CDN
# TYPE promitor_demo_redis_enterprise_memory_declared gauge
promitor_demo_redis_enterprise_memory_declared{tenant_id="e0372f7f-a362-47fb-9631-74a5c4ba8bbf",subscription_id="63c590b6-4947-4898-92a3-cae91a31b5e4",resource_uri="subscriptions/63c590b6-4947-4898-92a3-cae91a31b5e4/resourceGroups/promitor-testing-infrastructure-us/providers/Microsoft.Cache/RedisEnterprise/promitor-testing-resource-us-redis-enterprise",resource_group="promitor-testing-infrastructure-us",instance_name="promitor-testing-resource-us-redis-enterprise",geo="china",environment="dev"} 1 1639637837966
# HELP promitor_demo_redis_enterprise_memorys_discovered Amount of requests at the origin of our Azure CDN
# TYPE promitor_demo_redis_enterprise_memorys_discovered gauge
promitor_demo_redis_enterprise_memorys_discovered{tenant_id="e0372f7f-a362-47fb-9631-74a5c4ba8bbf",subscription_id="63c590b6-4947-4898-92a3-cae91a31b5e4",resource_uri="subscriptions/63c590b6-4947-4898-92a3-cae91a31b5e4/resourceGroups/promitor-testing-infrastructure-us/providers/Microsoft.Cache/RedisEnterprise/promitor-testing-resource-us-redis-enterprise",resource_group="promitor-testing-infrastructure-us",instance_name="promitor-testing-resource-us-redis-enterprise",geo="china",environment="dev"} 1 1639637885692
# HELP promitor_demo_maria_db_cpu_declared CPU usage of Maria DB servers
# TYPE promitor_demo_maria_db_cpu_declared gauge
promitor_demo_maria_db_cpu_declared{tenant_id="e0372f7f-a362-47fb-9631-74a5c4ba8bbf",subscription_id="63c590b6-4947-4898-92a3-cae91a31b5e4",resource_uri="subscriptions/63c590b6-4947-4898-92a3-cae91a31b5e4/resourceGroups/promitor-testing-infrastructure-us/providers/Microsoft.DBforMariaDB/servers/promitor-testing-resource-us-mariadb-server",resource_group="promitor-testing-infrastructure-us",instance_name="promitor-testing-resource-us-mariadb-server",geo="china",environment="dev"} 0.018 1639637903485
# HELP promitor_demo_maria_db_cpu_discovered CPU usage of Maria DB servers
# TYPE promitor_demo_maria_db_cpu_discovered gauge
promitor_demo_maria_db_cpu_discovered{tenant_id="e0372f7f-a362-47fb-9631-74a5c4ba8bbf",subscription_id="63c590b6-4947-4898-92a3-cae91a31b5e4",resource_uri="subscriptions/63c590b6-4947-4898-92a3-cae91a31b5e4/resourceGroups/promitor-testing-infrastructure-us/providers/Microsoft.DBforMariaDB/servers/promitor-testing-resource-us-mariadb-server",resource_group="promitor-testing-infrastructure-us",instance_name="promitor-testing-resource-us-mariadb-server",geo="china",environment="dev"} 0.018 1639637885691
# HELP promitor_demo_load_balancers_requests_declared Amount of requests at the origin of our Azure CDN
# TYPE promitor_demo_load_balancers_requests_declared gauge
promitor_demo_load_balancers_requests_declared{tenant_id="e0372f7f-a362-47fb-9631-74a5c4ba8bbf",subscription_id="63c590b6-4947-4898-92a3-cae91a31b5e4",resource_uri="subscriptions/63c590b6-4947-4898-92a3-cae91a31b5e4/resourceGroups/mc_tom-kerkhove-open-source-kubernetes_tom-kerkhove-open-source_westeurope/providers/Microsoft.Network/loadBalancers/kubernetes",resource_group="mc_tom-kerkhove-open-source-kubernetes_tom-kerkhove-open-source_westeurope",instance_name="kubernetes",geo="china",environment="dev"} 12627870 1639639986204
# HELP promitor_demo_load_balancers_requests_discovered Amount of requests at the origin of our Azure CDN
# TYPE promitor_demo_load_balancers_requests_discovered gauge
promitor_demo_load_balancers_requests_discovered{tenant_id="e0372f7f-a362-47fb-9631-74a5c4ba8bbf",subscription_id="63c590b6-4947-4898-92a3-cae91a31b5e4",resource_uri="subscriptions/63c590b6-4947-4898-92a3-cae91a31b5e4/resourceGroups/mc_tom-kerkhove-open-source-kubernetes_tom-kerkhove-open-source_westeurope/providers/Microsoft.Network/loadBalancers/kubernetes",resource_group="mc_tom-kerkhove-open-source-kubernetes_tom-kerkhove-open-source_westeurope",instance_name="kubernetes",geo="china",environment="dev"} 12627870 1639639987115
# HELP promitor_demo_vnet_ddos_declared Amount of requests at the origin of our Azure CDN
# TYPE promitor_demo_vnet_ddos_declared gauge
promitor_demo_vnet_ddos_declared{tenant_id="e0372f7f-a362-47fb-9631-74a5c4ba8bbf",subscription_id="63c590b6-4947-4898-92a3-cae91a31b5e4",resource_uri="subscriptions/63c590b6-4947-4898-92a3-cae91a31b5e4/resourceGroups/promitor-testing-infrastructure-us/providers/Microsoft.Network/virtualNetworks/promitor-testing-resource-us-vnet",resource_group="promitor-testing-infrastructure-us",instance_name="promitor-testing-resource-us-vnet",geo="china",environment="dev"} -1 1639639982439
# HELP promitor_demo_vnet_ddos_discovered Amount of requests at the origin of our Azure CDN
# TYPE promitor_demo_vnet_ddos_discovered gauge
promitor_demo_vnet_ddos_discovered{tenant_id="e0372f7f-a362-47fb-9631-74a5c4ba8bbf",subscription_id="63c590b6-4947-4898-92a3-cae91a31b5e4",resource_uri="subscriptions/63c590b6-4947-4898-92a3-cae91a31b5e4/resourceGroups/promitor-kubernetes-landscape/providers/Microsoft.Network/virtualNetworks/promitor-kubernetes-landscape-vnet",resource_group="promitor-kubernetes-landscape",instance_name="promitor-kubernetes-landscape-vnet",geo="china",environment="dev"} -1 1639639983712

Discovery output (CDN):

[{
		"$type": "Promitor.Core.Contracts.ResourceTypes.CdnResourceDefinition, Promitor.Core.Contracts",
		"CdnName": "promitor-testing-resource-us-cdn",
		"ResourceType": "Cdn",
		"SubscriptionId": "63c590b6-4947-4898-92a3-cae91a31b5e4",
		"ResourceGroupName": "promitor-testing-infrastructure-us",
		"ResourceName": "promitor-testing-resource-us-cdn",
		"UniqueName": "promitor-testing-resource-us-cdn"
	}
]

Discovery output (Cache for Redis Enterprise):

[{
	"$type": "Promitor.Core.Contracts.ResourceTypes.RedisEnterpriseCacheResourceDefinition, Promitor.Core.Contracts",
	"CacheName": "promitor-testing-resource-us-redis-enterprise",
	"ResourceType": "RedisEnterpriseCache",
	"SubscriptionId": "63c590b6-4947-4898-92a3-cae91a31b5e4",
	"ResourceGroupName": "promitor-testing-infrastructure-us",
	"ResourceName": "promitor-testing-resource-us-redis-enterprise",
	"UniqueName": "promitor-testing-resource-us-redis-enterprise"
}]

Discovery output (Database for MariaDB):

[{
	"$type": "Promitor.Core.Contracts.ResourceTypes.MariaDbResourceDefinition, Promitor.Core.Contracts",
	"ServerName": "promitor-testing-resource-us-mariadb-server",
	"ResourceType": "MariaDb",
	"SubscriptionId": "63c590b6-4947-4898-92a3-cae91a31b5e4",
	"ResourceGroupName": "promitor-testing-infrastructure-us",
	"ResourceName": "promitor-testing-resource-us-mariadb-server",
	"UniqueName": "promitor-testing-resource-us-mariadb-server"
}]

Discovery output (Load Balancer):

[{
	"$type": "Promitor.Core.Contracts.ResourceTypes.LoadBalancerResourceDefinition, Promitor.Core.Contracts",
	"LoadBalancerName": "kubernetes",
	"ResourceType": "LoadBalancer",
	"SubscriptionId": "63c590b6-4947-4898-92a3-cae91a31b5e4",
	"ResourceGroupName": "mc_tom-kerkhove-open-source-kubernetes_tom-kerkhove-open-source_westeurope",
	"ResourceName": "kubernetes",
	"UniqueName": "kubernetes"
}]

Discovery output (Virtual Network):

[{
	"$type": "Promitor.Core.Contracts.ResourceTypes.VirtualNetworkResourceDefinition, Promitor.Core.Contracts",
	"VirtualNetworkName": "promitor-kubernetes-landscape-vnet",
	"ResourceType": "VirtualNetwork",
	"SubscriptionId": "63c590b6-4947-4898-92a3-cae91a31b5e4",
	"ResourceGroupName": "promitor-kubernetes-landscape",
	"ResourceName": "promitor-kubernetes-landscape-vnet",
	"UniqueName": "promitor-kubernetes-landscape-vnet"
}]

Fixes #1795
Fixes #1880
Fixes #1664
Fixes #1845
Fixes #1170

@tomkerkhove
Provide configuration
708079d 
Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>
@tomkerkhove
Add serialization tests
d7b85ba 
Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>
@tomkerkhove
Add resource discovery
3e6d1c6 
Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>
@tomkerkhove
Add validation
7492cf8 
Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>
@tomkerkhove
Provide scrapers
0895bd4 
Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>
@trafico-bot trafico-bot bot added the 🔍 Ready for Review Pull Request is not reviewed yet label Dec 14, 2021
@tomkerkhove tomkerkhove mentioned this pull request Dec 16, 2021
Merged
@tomkerkhove
Provide changelog entries
0e59072 
Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>
@tomkerkhove
Provide documentation
13baaba 
Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>
@tomkerkhove tomkerkhove marked this pull request as ready for review December 17, 2021 06:51
@tomkerkhove
Merge branch 'master' into moar-scrapers
deb4e2b
@tomkerkhove
Add missing description and leave out Atlassian for local testing
c9a9ddc 
Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>
@tomkerkhove
Provide note on CDN metrics
9d2ea83 
Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>
@tomkerkhove tomkerkhove merged commit 9ee0930 into master Dec 17, 2021
@tomkerkhove tomkerkhove deleted the moar-scrapers branch December 17, 2021 09:28
@trafico-bot trafico-bot bot added ✨ Merged Pull Request has been merged successfully and removed 🔍 Ready for Review Pull Request is not reviewed yet labels Dec 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
✨ Merged Pull Request has been merged successfully
Projects
None yet
Milestone
No milestone
1 participant
@tomkerkhove