sapcc · thgrs · Oct 23, 2019 · Sep 6, 2019 · Sep 12, 2019 · Sep 12, 2019
diff --git a/.travis.yml b/.travis.yml
@@ -2,7 +2,6 @@ language: go
 sudo: false
 
 go:
-    - 1.11
     - 1.12
 
 install:

diff --git a/.vscode/launch.json b/.vscode/launch.json
@@ -0,0 +1,33 @@
+{
+    // Use IntelliSense to learn about possible attributes.
+    // Hover to view descriptions of existing attributes.
+    // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
+    "version": "0.2.0",
+    "configurations": [
+        {
+            "name": "Launch CLI",
+            "type": "go",
+            "request": "launch",
+            "args": ["query", "--format", "table", "--columns", "domain", "limes_project_quota"],
+            "mode": "debug",
+            "program": "${workspaceFolder}/main.go",
+            "envFile": "${workspaceFolder}/.env"
+        },
+        {
+            "name": "Launch test package",
+            "type": "go",
+            "request": "launch",
+            "mode": "test",
+            "program": "${file}"
+        },
+        {
+            "name": "Server",
+            "type": "go",
+            "request": "launch",
+            "mode": "auto",
+            "program": "${workspaceRoot}/main.go",
+            "env": { "MAIA_DEBUG": "1" },
+            "args": [ "serve" ]
+        }
+    ]
+}
diff --git a/docs/developers-guide.md b/docs/developers-guide.md
@@ -68,6 +68,17 @@ The problem with basic authentication is that it lacks a standard way to express
  denoted by name: `projectname@domainname`. To disambiguate scoping by project-id and domain-name, the domain is always prefixed
  with `@`.
 
+ #### Application Credential Authentication
+
+ Maia also supports authentication with application credentials.  In order to use application credentials you must supply an application credential secret in conjunction with either an application credential ID, or an application credential name and an openstack username.  
+
+ The scope of the authentication for application credentials will be determined by the project in which the application credentials were created.
+
+ Application credential authentication is also supported through the following header fields:
+     'X-Application-Credential-Id'
+     'X-Application-Credential-Name'
+     'X-Application-Credential-Secret'
+
 #### Variants
 
 This scheme expands into five variants to express username and authorization scope:
@@ -143,13 +154,29 @@ Components/Packages
 * api: Implementation of the API
 * cmd: Implementation of the CLI
 * keystone: authentication plugin(s)
-* prometheus: glue code to attach to a Prometheus as storage
+* storage: glue code to attach to a Prometheus as storage
+* test: helper classes for tests
+* ui: the Prometheus 1.x UI adapted to Maia
+* util: helper classes
 
-The latter packages will probably be renamed if we decide to support additional user
-management services or other monitoring backends (e.g. project cortex).
+The _keystone_ and _storage_ package contain some preparational work to plug-in other backends for authentication and authorization as well as metric storage. This is good enough to plug-in dummy implementations for testing purposes and it helps preventing uncontrolled growths of dependency. For real pluggability of alternative backend services the currently level of abstraction is not enough though.
 
 ![Architecture diagram](./maia-architecture.png)
 
+## Building
+
+Travis is used for building, static code checks and build-verification testing ([tutorial](https://docs.travis-ci.com/user/tutorial/)). No real integration testing takes place here.
+
+The _make_ file contains additional targets that can be used e.g. to build continuous deployment pipelines:
+* all: builds and tests the binary for the current platform from scratch
+* clean: removes temporary files, vendored go packages dependencies and generated code
+* generate: recreates generated code (mocks, UI binding code)
+* build: builds the executable for the current platfrom
+* build/platforms: builds the executable for all supported platforms
+* build/docker: builds a docker image
+* vendor: updates the vendored go packages to latest (according to `glide.yaml`)
+* check: runs static source code checks and executes unit tests
+
 ## Links
 
 [![Build Status](https://travis-ci.org/sapcc/maia.svg?branch=master)](https://travis-ci.org/sapcc/maia)

diff --git a/docs/operators-guide.md b/docs/operators-guide.md
@@ -37,7 +37,7 @@ First you need to decide which port the Maia service should listen to.
 bind_address = "0.0.0.0:9091"
 ```
 
-### Prometheus
+### Prometheus/Thanos
 
 Any data served by Maia is served by the underlying Prometheus installation that acts as a TSDB and data collection layer.
 
@@ -48,6 +48,16 @@ prometheus_url = "http://myprometheus:9090"
 # proxy = proxy for reaching <prometheus_url>
 ```
 
+If you use _Thanos_, you have to provide the Thanos _querier_ address in the `prometheus_url`. Since the _querier_ does not
+support the `federate` API [yet](https://github.com/thanos-io/thanos/pull/1546), you have specify another parameter `federate_url` which should point to an endpoint that supports it.
+Usually this will be a Prometheus that has recent datapoints for _all_ the series the _querier_ knows (except for stale ones).
+
+```
+prometheus_url = "http://mythanos:9090/thanos"
+federate_url = "http://myprometheus:9090"
+# proxy = proxy for reaching <prometheus_url>
+```
+
 ### Performance
 
 The Prometheus API does not offer an efficient way to list known all historic label values for a given tenant. This

diff --git a/docs/users-guide.md b/docs/users-guide.md
@@ -48,14 +48,25 @@ URL: https://maia.myopenstack.net
 Username: myUser
 ```
 
-You may also use the special username syntax described [here](#openstack-authentication-and-authorization)
+You may also use the special username syntax described in more detail [here](#openstack-authentication-and-authorization)
 to log right into your target project.
 
 ```
-Username: myUser@mydomain|myproject@mydomain
+Username: myuser@mydomain|myproject@mydomain
 Password: ********
 ```
 
+Or you use OpenStack _application credentials_:
+
+```
+# this is an example of ID-based login
+username: *myappcredid
+password: myappcredsecret
+# this is an example of name-based login
+username: *myappcredname@myuser@mydomain
+password: myappcredsecret
+```
+
 ### The Maia Screen
 
 The Maia screen consists of three part:
@@ -111,14 +122,18 @@ The `maia` command can also be used to retrieve metrics from the Maia service. I
 | --os-username | OS_USERNAME | OpenStack username, requires `os-user-domain-name` |
 | --os-user-id | OS_USER_ID | OpenStack user unique ID |
 | --os-password | OS_PASSWORD | Password |
+| --os-token | OS_TOKEN | Pregenerated Keystone token with authorization scope |
+| --os-application-credential-id | OS_APPLICATION_CREDENTIAL_ID | ID of an _application credential_ |
+| --os-application-credential-name | OS_APPLICATION_CREDENTIAL_NAME | name of an _application credential_, scoped by user |
+| --os-application-credential-secret | OS_APPLICATION_CREDENTIAL_SECRET | secret of an _application credential_ |
 | --os-user-domain-name | OS_USER_DOMAIN_NAME | domain name, qualifying the username (default: `Default`) |
 | --os-user-domain-id | OS_USER_DOMAIN_ID | domain unique ID, qualifying the username (default: `default`) |
 | --os-project-name | OS_PROJECT_NAME | OpenStack project name for authorization scoping to project, requires `os-project-domain-name` |
 | --os-project-id | OS_PROJECT_ID | OpenStack project unique ID |
 | --os-domain-name | OS_DOMAIN_NAME | OpenStack domain name for authorization scoping to domain |
 | --os-domain-id | OS_DOMAIN_ID | OpenStack domain unique ID for authorization scoping to domain |
-| --os-token | OS_TOKEN | Pregenerated Keystone token with authorization scope |
 | --os-auth-url | OS_AUTH_URL | Endpoint of the Identity v3 service. Needed to authentication and Maia endpoint lookup |
+| --os-auth-type | OS_AUTH_TYPE | Authentication method to use: one of `password`, `token`, `v3applicationcredential`|
 
 Usually, you can reuse your existing RC-files. For performance reasons, you should consider token-based
 authentication whenever you make several calls to the Maia CLI.
@@ -283,7 +298,13 @@ Domain scoped user:
 * `user_id|@domain_name`
 * `user_name@user_domain_name|@domain_name`
 
-### Background: OpenStack Authentication and Authorization
+Application Credential:
+
+* `*app_cred_id`
+* `*app_cred_name@user_id`
+* `*app_cred_name@user_name@user_domain_name`
+
+### OpenStack Authentication and Authorization
 
 In addition to 'native' OpenStack authentication using Keystone tokens, Maia supports basic authentication in order
 to support existing clients like Grafana and federated Prometheus.
@@ -301,6 +322,18 @@ The problem with basic authentication is that it lacks a standard way to express
  denoted by name: `projectname@domainname`. To disambiguate scoping by project-id and domain-name, the domain is always prefixed
  with `@`.
 
+Alternatively, OpenStack _application credentials_ can be used in place of username and password. With these credentials you are implicitly scoped
+to a single project (or domain), so there is no need to supply scope information as before.
+
+To tell Maia that the username and password fields are actually containing _application credentials_,
+you put an asterisk (`*`) in front of the username value.
+
+There are two ways to authenticate with application credentials:
+* ID-based: Use the application credential ID as username
+* Name-based: Use the application credential name and qualify it using the username or user ID
+
+In both cases you use the _secret_ of the application credential as password.
+
 # Federating Maia to Prometheus
 
 To configure Prometheus to receive data from Maia, the following job configuration has to be applied.

diff --git a/pkg/api/api_test.go b/pkg/api/api_test.go
@@ -25,7 +25,8 @@ import (
 	"testing"
 
 	"errors"
-	"github.com/databus23/goslo.policy"
+
+	policy "github.com/databus23/goslo.policy"
 	"github.com/golang/mock/gomock"
 	"github.com/gophercloud/gophercloud/openstack/identity/v3/tokens"
 	"github.com/prometheus/client_golang/prometheus"
@@ -235,12 +236,12 @@ func TestLabelValues(t *testing.T) {
 	expectAuthByProjectID(keystoneMock)
 	// Maia's label-values implementation uses the series API and a time-based filter stale series out. The exact start
 	// and end date of the filter cannot be predicted, therefore we accept anything that is a parsable date.
-	storageMock.EXPECT().Series([]string{"{component!=\"\",project_id=\"12345\"}"}, test.TimeStringMatcher{}, test.TimeStringMatcher{}, storage.JSON).Return(test.HTTPResponseFromFile("fixtures/series.json"), nil)
+	storageMock.EXPECT().QueryRange("count({project_id=\"12345\",service!=\"\"}) BY (service)", test.TimeStringMatcher{}, test.TimeStringMatcher{}, viper.Get("maia.label_value_ttl"), "", storage.JSON).Return(test.HTTPResponseFromFile("fixtures/label_values_query_range.json"), nil)
 
 	test.APIRequest{
 		Headers:          map[string]string{"Authorization": base64.StdEncoding.EncodeToString([]byte("Basic user_id|12345:password")), "Accept": storage.JSON},
 		Method:           "GET",
-		Path:             "/api/v1/label/component/values",
+		Path:             "/api/v1/label/service/values",
 		ExpectStatusCode: http.StatusOK,
 		ExpectJSON:       "fixtures/label_values.json",
 	}.Check(t, router)

diff --git a/pkg/api/fixtures/label_values_query_range.json b/pkg/api/fixtures/label_values_query_range.json
@@ -0,0 +1,101 @@
+{
+    "status": "success",
+    "data": {
+        "resultType": "matrix",
+        "result": [
+            {
+                "metric": {
+                    "service": "dns"
+                },
+                "values": [
+                    [
+                        1567641600,
+                        "20"
+                    ],
+                    [
+                        1568160000,
+                        "10"
+                    ]
+                ]
+            },
+            {
+                "metric": {
+                    "service": "barbican"
+                },
+                "values": [
+                    [
+                        1567641600,
+                        "5"
+                    ],
+                    [
+                        1568160000,
+                        "5"
+                    ]
+                ]
+            },
+            {
+                "metric": {
+                    "service": "nova"
+                },
+                "values": [
+                    [
+                        1567641600,
+                        "2"
+                    ]
+                ]
+            },
+            {
+                "metric": {
+                    "service": "neutron"
+                },
+                "values": [
+                    [
+                        1567641600,
+                        "105"
+                    ]
+                ]
+            },
+            {
+                "metric": {
+                    "service": "objectstore"
+                },
+                "values": [
+                    [
+                        1568160000,
+                        "105"
+                    ]
+                ]
+            },
+            {
+                "metric": {
+                    "service": "keystone"
+                },
+                "values": [
+                    [
+                        1567641600,
+                        "4"
+                    ],
+                    [
+                        1568160000,
+                        "4"
+                    ]
+                ]
+            },
+            {
+                "metric": {
+                    "service": "glance"
+                },
+                "values": [
+                    [
+                        1567641600,
+                        "1"
+                    ],
+                    [
+                        1568160000,
+                        "9"
+                    ]
+                ]
+            }
+        ]
+    }
+}