Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Server side implementation for the data warehouse feature #19490

Merged
merged 21 commits into from
Jun 26, 2024
Merged

Server side implementation for the data warehouse feature #19490

merged 21 commits into from
Jun 26, 2024

Conversation

ousmaneo
Copy link
Contributor

@ousmaneo ousmaneo commented May 29, 2024 •
edited by kodjo-anipah
Loading

/jpd Graylog2/graylog-plugin-enterprise#7208
/nocl
Server-side PR for implementing a data warehouse feature in Graylog.
Changes:

  • added relevant data warehouse metrics to the default prometheus-exporter.yml
  • added a new StreamsDetails page, for controlling all stream relevant actions and settings (e.g stream rules, pipelines...)
  • added Stream guard concept, to add the capability to prevent stream deletions in the enterprise plugin

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Refactoring (non-breaking change)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes.

@ousmaneo ousmaneo changed the title Data warehouse Add data warehouse prometheus exporter Jun 14, 2024
@ousmaneo ousmaneo marked this pull request as ready for review June 14, 2024 06:09
@kodjo-anipah kodjo-anipah changed the title Add data warehouse prometheus exporter Add data warehouse feature Jun 19, 2024
kodjo-anipah and others added 9 commits June 21, 2024 15:55
@kodjo-anipah @ousmaneo @bernd
introduced concept of stream guard to throw an exception on invalid s… (
9508dfc 
#19454)

* introduced concept of stream guard to throw an exception on invalid stream deletion requests

* Implement Stream details page (#19342)

* add api to get all connected piplines for a stream id

* Add stream details page

- add Stream pipeline list

* update StreamDetailsPage

* add alignActionsLeft to Section

* add stream dataWarehouse

- update design

* fix StreamResource.java

* add Mantine Text component

* update StreamOverview

- add TitleCell
- hide description
- update styles

* Update stream details

- fix linter error

* fix linter

* fix StreamOverview test

* fix linters

* fix linter

* fix review

* fix description

* disable details button for not editable stream

---------

Co-authored-by: Maxwell Anipah <maxwell.anipah@graylog.com>
Co-authored-by: Maxwell <98284293+kodjo-anipah@users.noreply.github.com>

* move stream guard to service class and add test

* move out streamGuardException inner class

* add license header

* clean up

* remove unused import

---------

Co-authored-by: Ousmane SAMBA <ousmane@graylog.com>
Co-authored-by: Bernd Ahlers <bernd@users.noreply.github.com>
Co-authored-by: Ousmane Samba <ousmanesamba@gmail.com>
@bernd
Merge branch 'master' into cold-tier
d1fbc0c
@bernd
Add PluginModule#addStreamDeletion guard method
871cf16
@ousmaneo
Merge branch 'master' into cold-tier
3747359
@kodjo-anipah
Merge branch 'master' into cold-tier
99bcc3f
@kodjo-anipah
Merge branch 'master' into cold-tier
51edbb6
@ousmaneo @kodjo-anipah
@bernd @gally47
Add stream details Index set section (#19640)
ddb658c 
* introduced concept of stream guard to throw an exception on invalid stream deletion requests

* Implement Stream details page (#19342)

* add api to get all connected piplines for a stream id

* Add stream details page

- add Stream pipeline list

* update StreamDetailsPage

* add alignActionsLeft to Section

* add stream dataWarehouse

- update design

* fix StreamResource.java

* add Mantine Text component

* update StreamOverview

- add TitleCell
- hide description
- update styles

* Update stream details

- fix linter error

* fix linter

* fix StreamOverview test

* fix linters

* fix linter

* fix review

* fix description

* disable details button for not editable stream

---------

Co-authored-by: Maxwell Anipah <maxwell.anipah@graylog.com>
Co-authored-by: Maxwell <98284293+kodjo-anipah@users.noreply.github.com>

* move stream guard to service class and add test

* move out streamGuardException inner class

* add license header

* clean up

* add useSingleIndexSet hook

* add indexSetDestitation section and actions

* add bottom margin to Section

* fix linter

* remove unused import

* fix reviews

---------

Co-authored-by: Maxwell Anipah <maxwell.anipah@graylog.com>
Co-authored-by: Maxwell <98284293+kodjo-anipah@users.noreply.github.com>
Co-authored-by: Bernd Ahlers <bernd@users.noreply.github.com>
Co-authored-by: Mohamed OULD HOCINE <106236152+gally47@users.noreply.github.com>
@ousmaneo @kodjo-anipah
@bernd @gally47
Add stream pipeline connection action (#19587)
a4c92ac 
* introduced concept of stream guard to throw an exception on invalid stream deletion requests

* Implement Stream details page (#19342)

* add api to get all connected piplines for a stream id

* Add stream details page

- add Stream pipeline list

* update StreamDetailsPage

* add alignActionsLeft to Section

* add stream dataWarehouse

- update design

* fix StreamResource.java

* add Mantine Text component

* update StreamOverview

- add TitleCell
- hide description
- update styles

* Update stream details

- fix linter error

* fix linter

* fix StreamOverview test

* fix linters

* fix linter

* fix review

* fix description

* disable details button for not editable stream

---------

Co-authored-by: Maxwell Anipah <maxwell.anipah@graylog.com>
Co-authored-by: Maxwell <98284293+kodjo-anipah@users.noreply.github.com>

* move stream guard to service class and add test

* add stream pipeline connection

* add changelog

* update changelog type

* move changelog to right folder

* move out streamGuardException inner class

* add license header

* add initial value for pipelines

* invalid pipeline query after save

* fix review

* make sure current pipeline is set

* sort connected pipeline

* revert merge errror

---------

Co-authored-by: Maxwell Anipah <maxwell.anipah@graylog.com>
Co-authored-by: Maxwell <98284293+kodjo-anipah@users.noreply.github.com>
Co-authored-by: Bernd Ahlers <bernd@users.noreply.github.com>
Co-authored-by: Mohamed OULD HOCINE <106236152+gally47@users.noreply.github.com>
@bernd
Add StreamService#loadStreamTitles
6e4d422
@bernd bernd mentioned this pull request Jun 25, 2024
Merged
@kodjo-anipah kodjo-anipah self-assigned this Jun 25, 2024
@kodjo-anipah kodjo-anipah changed the title Add data warehouse feature Server side implementation for the data warehouse feature Jun 26, 2024
@bernd bernd merged commit 2964366 into master Jun 26, 2024
12 checks passed
@bernd bernd deleted the cold-tier branch June 26, 2024 09:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bernd bernd bernd approved these changes

Assignees

@kodjo-anipah kodjo-anipah

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@ousmaneo @bernd @kodjo-anipah