Skip to content

Commit

Permalink
convert the RFC to a README
Browse files Browse the repository at this point in the history
  • Loading branch information
@pmuellr
pmuellr committed Dec 13, 2019
1 parent 0753e46 commit 9abc912
Showing 1 changed file with 99 additions and 126 deletions.
225 changes: 99 additions & 126 deletions rfcs/text/0007_event_log.md → x-pack/plugins/event_log/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,12 +1,69 @@
- Start Date: 2019-09-12
- RFC PR: currently https://github.com/elastic/kibana/pull/45081
- Kibana Issue: (leave this empty)
# Event Log

See also issue https://github.com/elastic/kibana/issues/45083, which is an
umbrella issue tracking ongoing initial work.
## Overview

The purpose of this plugin is to provide a way to persist a history of events
occuring in Kibana, initially just for the Make It Action project - alerts
and actions.

# Summary

## Basic Usage - Logging Events

Follow these steps to use `event_log` in your plugin:

1. Declare `event_log` as a dependency in `kibana.json`:

```json
{
...
"requiredPlugins": ["event_log"],
...
}
```

2. Register provider / actions, and create your plugin's logger, using service
API provided in the `setup` stage:

```typescript
...
import { IEventLogger, IEventLogService } from '../../event_log/server';
interface PluginSetupDependencies {
event_log: IEventLogService;
}
...
public setup(core: CoreSetup, { event_log }: PluginSetupDependencies) {
...
event_log.registerProviderActions('my-plugin', ['action-1, action-2']);
const eventLogger: IEventLogger = event_log.getLogger({ event: { provider: 'my-plugin' } });
...
}
...
```

4. To log an event, call `logEvent()` on the `eventLogger` object you created:

```typescript
...
eventLogger.logEvent({ event: { action: 'action-1' }, tags: ['fe', 'fi', 'fo'] });
...
```


## Testing

### Unit tests

From `kibana-root-folder/x-pack`, run:
```bash
$ node node scripts/jest plugins/event_log
```

### API Integration tests

None yet!


## Background

For the Make It Action alerting / action plugins, we will need a way to
persist data regarding alerts and actions, for UI and investigative purposes.
Expand Down Expand Up @@ -85,27 +142,25 @@ Here's the event written to the event log index:
}
```

The shape of the document written to the index is a subset of [ECS][] with an
extended field of `kibana` with some Kibana-related properties contained within
it.

# Motivation
The ES mappings for the ECS data, and the config-schema for the ECS data, are
generated by a script, and available here:

The existing designs for Make It Action describe UIs which show some amount
of alert "history". Up till now, we had no where to write this history,
and so a new elasticsearch index is required for that. Because we already
have two known clients of this plugin, which are related but different,
the design of the event documents is very generic, which means it could easily
be used by other plugins that would like to record "events", for the same
general purposes.
- [`generated/mappings.json`](generated/mappings.json)
- [`generated/schemas.ts`](generated/schemas.ts)

The shape of the document written to the index is [ECS][] with an extended field
of `kibana` with some Kibana-related properties contained within it.
It's anticipated that these interfaces will grow over time, hopefully adding
more ECS fields but adding Kibana extensions as required.

Since there are some security concerns with the data, we are currently
restricting access via known saved object ids. That is, you can only query
history records associated with specific saved object ids.

[ECS]: https://www.elastic.co/guide/en/ecs/current/index.html

# Detailed design

## API

Expand All @@ -123,6 +178,8 @@ export interface IEventLogService {

export interface IEventLogger {
logEvent(properties: IEvent): void;
startTiming(event: IEvent): void;
stopTiming(event: IEvent): void;
}
```

Expand Down Expand Up @@ -159,6 +216,27 @@ method, nor what it would do if the method threw an error. All the error
processing involved with getting the data into the index is handled internally,
and logged to the server log as appropriate.

The `startTiming()` and `stopTiming()` methods can be used to set the timing
properties `start`, `end`, and `duration` in the event. For example:

```typescript
const loggedEvent: IEvent = { event: { action: 'foo' } };

// sets event.start
eventLogger.startTiming(loggedEvent);

longRunningFunction();

// sets event.end and event.duration
eventLogger.stopTiming(loggedEvent);

eventLogger.logEvent(loggedEvent);

```

It's anticipated that more "helper" methods like this will be provided in the
future.


## Stored data

Expand All @@ -168,8 +246,9 @@ with older event documents deleted, turned cold, frozen, etc. We'll supply
some default values, but customers will be able to tweak these.

The index template, mappings, config-schema types, etc for the index can
be found in the [x-pack/plugins/event_log/generated directory](../../x-pack/plugins/event_log/generated). These files are generated from a script which takes as
input the ECS properties to use, and the Kibana extensions.
be found in the [generated directory](generated). These files are generated
from a script which takes as input the ECS properties to use, and the Kibana
extensions.

See [ilm rollover action docs][] for more info on the `is_write_index`, and `index.lifecycle.*` properties.

Expand All @@ -182,6 +261,7 @@ Of particular note in the `mappings`:

We may change some of that before releasing.


## ILM setup

We'll want to provide default ILM policy, this seems like a reasonable first
Expand Down Expand Up @@ -217,110 +297,3 @@ For more relevant information on ILM, see:
[getting started with ILM doc]: https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html
[write index alias behavior]: https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-rollover-index.html#indices-rollover-is-write-index


# Drawbacks

Leaves things up to the user to customize the ILM policy directly.


# Alternatives

## Using Saved Objects instead of a new index

See [issue 51223](https://github.com/elastic/kibana/issues/51223) for a
discussion on whether to try to use saved objects as the primary store for
event log documents, versus using a new stand-alone index.

Summary: we'll use a new stand-alone index, but use the "link" saved object
pattern to only allow access via a saved object. Ie, you will need to have
access to a saved object, to get it's id, to then use in a search through
the event log index.

## Concern with overlap with eventual audit log capability

In an early review, concern was raised that there is a lot of overlap with
where we eventually be with audit logging.

The issues with that are:

- audit logs are currently just an `audit-log` tag on messages written to the
standard Kibana log; there is no infrastructure to allow them to be queried

- security differences; the users able to read audit logs are probably much
more constrained than users that need to be able to read these event logs

So, we don't really have an _alternative_ here, but ... a lot of overlap.
Long-term, it may make sense to make the eventual audit logging code
parameterizable, so that the core code could be used for other purposes, like
this event log. We're just not there yet. Or implement the audit log with
the event log :-)

## Write to a events to a file and ingest via FileBeat

This came up as _"why don't we use our own stuff to do this since that's what
it's designed for"_. Yup! However, the only thing that really changes over
writing our index is:

- we'd write to some other log (could be console.log, or a file) instead of
an index, directly
- we'd have to arrange to run FileBeat alongside Kibana, ingesting these
logged events, which would send them into elasticsearch

We still need to arrange to initialize all the elasticsearch resources (ilm,
index template, etc).

The big kicker though is: where would we right these that FileBeat could
find them? Writing to a file might be problematic for users that run
Kibana with no writable file system, including folks using docker-based
deployments.

Lots of unknown unknowns, it seems.

Interestingly enough, this kind of story might work out extremely well for
Elastic Cloud, where we have a lot more control over the deployments!

# Adoption strategy

If we implement this proposal, how will existing Kibana developers adopt it? Is
this a breaking change? Can we write a codemod? Should we coordinate with
other projects or libraries?

The API surface has been kept to a minimum to make it easy for other plugins
make use of it. But that's just a nice-to-have right now anyway, since
the only known clients will be actions and alerting.


# How we teach this

**What names and terminology work best for these concepts and why? How is this
idea best presented? As a continuation of existing Kibana patterns?**

"event" and "log" are probably the most controversial names in use here.
Previously, we've referred to this as an "audit log" or "history log".
Audit log is not a great name due to the existance of the audit logging
facility already available in Kibana. History or History log seems like
a closer fit. Event log works for me, as it provides just a touch more
information than history - it's a history of events - but of course "events"
as suitable vague enough that it doesn'tt add thatt much more.

**Would the acceptance of this proposal mean the Kibana documentation must be
re-organized or altered? Does it change how Kibana is taught to new developers
at any level?**

no

**How should this feature be taught to existing Kibana developers?**

Follow examples of other plugins using it.

The API surface is very small, it's designed to be easy for other developers
to reuse.


# Unresolved questions

**Optional, but suggested for first drafts. What parts of the design are still
TBD?**

There is a task list of resolved and unresolved TODO items and future-looking
items in the initial event log PR https://github.com/elastic/kibana/issues/45083

0 comments on commit 9abc912

Please sign in to comment.