SWATCH-2021: Report filtered subscription data to Export Service in JSON

[Sgitario]

Jira issue: SWATCH-2021

Description

Added support of JSON uploads via export service.
I could verified e2e the integration with a real export-service, and I could spot that we were using the UUIDs wrongly, the ID was the Export Request UUID and the resource ID was the UUID.

Testing

1.- start kafka and postgresql: podman-compose up -d
2.- start export-service: podman-compose -f config/export-service/docker-compose.yml up -d
3.- start subscription sync service: DEV_MODE=true SPRING_PROFILES_ACTIVE="capacity-ingress,api,kafka-queue" ./gradlew :bootRun
4.- load some data:

INSERT INTO offering (sku, product_name, product_family, cores, sockets, hypervisor_cores, hypervisor_sockets, role, sla, usage, description, has_unlimited_usage, derived_sku, metered) VALUES ('MW02393', 'OpenShift Dedicated', 'OpenShift Enterprise', 0, 0, 0, 0, 'osd', 'Premium', 'Production', 'Red Hat OpenShift Service on AWS Hosted Control Planes (Hourly)', false, null, null);
INSERT INTO subscription (sku, org_id, subscription_id, quantity, start_date, end_date, billing_provider_id, subscription_number, billing_provider, billing_account_id) VALUES ('MW02393', '13259775', '13579593', 1, '2023-09-25 08:25:22.797423 +00:00', '2024-09-25 08:25:22.000000 +00:00', 'bcwuzc8ww10vvxfair55mliy8;4G5dKpAKahm', '13579750', 'aws', '505066801768');
INSERT INTO subscription_measurements (subscription_id, start_date, metric_id, measurement_type, value) VALUES ('13579593', '2023-09-25 08:25:22.797423 +00:00', 'Cores', 'PHYSICAL', 5);
INSERT INTO subscription_measurements (subscription_id, start_date, metric_id, measurement_type, value) VALUES ('13579593', '2023-09-25 08:25:22.797423 +00:00', 'Instance-hours', 'PHYSICAL', 1);

5.- create a new export request from Export Service:

get the identifier first:

echo -n '{"identity":{"account_number":"10001","org_id":"13259775","internal":{"org_id":"13259775"},"type":"User","user":{"username":"user_dev"}}}' | base64 -w 0
> eyJpZGVudGl0eSI6eyJhY2NvdW50X251bWJlciI6IjEwMDAxIiwib3JnX2lkIjoiMTMyNTk3NzUiLCJpbnRlcm5hbCI6eyJvcmdfaWQiOiIxMzI1OTc3NSJ9LCJ0eXBlIjoiVXNlciIsInVzZXIiOnsidXNlcm5hbWUiOiJ1c2VyX2RldiJ9fX0=

call the export service:

curl -sS -X POST http://localhost:8002/api/export/v1/exports -H "x-rh-identity: eyJpZGVudGl0eSI6eyJhY2NvdW50X251bWJlciI6IjEwMDAxIiwib3JnX2lkIjoiMTMyNTk3NzUiLCJpbnRlcm5hbCI6eyJvcmdfaWQiOiIxMzI1OTc3NSJ9LCJ0eXBlIjoiVXNlciIsInVzZXIiOnsidXNlcm5hbWUiOiJ1c2VyX2RldiJ9fX0=" -H "Content-Type: application/json" -d '{
    "name": "Test Export Request",
    "format": "json",
    "expires_at": "2025-01-01T00:00:00Z",
    "sources": [
        {
            "application": "subscriptions",
            "resource": "subscriptions",
            "filters": {
                "metric_id": "Cores"
            }
        }
    ]
}'

Expected output is:

{"id":"e81bc918-6434-4a58-9192-8a8dd73ec8fb","created_at":"2024-03-21T09:05:08.943615204Z","expires_at":"2025-01-01T00:00:00Z","name":"Test Export Request","format":"json","status":"pending","sources":[{"id":"f564ff70-9176-4413-ab6e-96a02c8ee124","application":"subscriptions","status":"pending","resource":"subscriptions","filters":{"metric_id":"Cores"}}]}

6.- verification:

check the subscription sync logs where you should see the new event coming from the Export service topic:

Processing event: e3ee74d4-ff1d-4373-9144-9ed8323258ed from application: urn:redhat:source:console:app:export-service
Event processed: e3ee74d4-ff1d-4373-9144-9ed8323258ed from application: urn:redhat:source:console:app:export-service

The export-service should have uploaded the generated report by the subscription sync service into S3 (minio instance).
You can verify that this report has been uploaded by checking the folder 'config/export-service/tmp/minio/exports-bucket' where should be a file at <BUCKET ID>.json/xl.meta. The xl.meta file is binary, but if you open the txt plain editor, you should see:

{"data":[{"sku":"MW02393","product_name":"OpenShift Dedicated","service_level":"Premium","usage":"Production","org_id":"13259775","subscription_number":"13579750","quantity":1,"measurements":[{"metric_id":"Cores","capacity":5.0,"measurement_type":"PHYSICAL"},{"metric_id":"Instance-hours","capacity":1.0,"measurement_type":"PHYSICAL"}]}]}

7.- verify wrong filters:

let's try to use a wrong value for product ID:

curl -sS -X POST http://localhost:8002/api/export/v1/exports -H "x-rh-identity: eyJpZGVudGl0eSI6eyJhY2NvdW50X251bWJlciI6IjEwMDAxIiwib3JnX2lkIjoiMTMyNTk3NzUiLCJpbnRlcm5hbCI6eyJvcmdfaWQiOiIxMzI1OTc3NSJ9LCJ0eXBlIjoiVXNlciIsInVzZXIiOnsidXNlcm5hbWUiOiJ1c2VyX2RldiJ9fX0=" -H "Content-Type: application/json" -d '{
    "name": "Test Export Request",
    "format": "json",
    "expires_at": "2025-01-01T00:00:00Z",
    "sources": [
        {
            "application": "subscriptions",
            "resource": "subscriptions",
            "filters": {
                "product_id": "Wrong!"
            }
        }
    ]
}'

Then, you should see the following traces in the the subscription sync logs:

2024-03-21 10:29:01,472 [thread=swatch-subscription-export-0-C-1] [INFO ] [org.candlepin.subscriptions.subscription.export.ExportSubscriptionListener] - Processing event: '887b5c96-01fb-4217-a222-1a52db890278' from application: 'urn:redhat:source:console:app:export-service'
2024-03-21 10:29:01,475 [thread=swatch-subscription-export-0-C-1] [ERROR] [org.candlepin.subscriptions.subscription.export.ExportSubscriptionListener] - Error thrown for event: '887b5c96-01fb-4217-a222-1a52db890278' sending ErrorRequest: 'Wrong filter in export request: ProductId: Wrong! not found in configuration'

And in the export-service database (by default, it's: "localhost:5342", user: postgres, password: postgres, database name: postgres), you can see a new record in the table "sources" with:

f9288f52-59dc-4ad6-9307-8c67de75c09c	fb177ea7-a8ea-43e4-8c2e-cb4bd7170601	subscriptions	failed	subscriptions	{"product_id": "Wrong!"}	400	ProductId: Wrong! not found in configuration

[ntkathole]

/retest

[san7ket]

/retest

[san7ket]

/retest

[ntkathole]

/retest

[san7ket]

@Sgitario In failed tests, I am seeing increase in time , for export request to be completed, with approx 220+ subs in account. What would be the expected wait_time here?
Testing are querying the status endpoint every 20 sec , 15 times currently. That is almost wait time of 5 mins

[san7ket]

/retest

[san7ket]

/retest