Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mapping configuration concept #129

Closed
de-jcup opened this issue Dec 18, 2019 · 4 comments
Closed

Mapping configuration concept #129

de-jcup opened this issue Dec 18, 2019 · 4 comments

Comments

@de-jcup
Copy link
Member

de-jcup commented Dec 18, 2019

Usecase

As an administrator I want to change my ScanConfig by a rest call at administration domain. All running instances of sechub shall be synched and use this configuration

Situation

With #127 we introduced a way to configure scan behavours.
This was done by @Value injection. This is okay as first attempt, but we need to restart server to have changes.

Also we got restrictions about configuration sizes (e.g. on Linux 128KB, see https://unix.stackexchange.com/questions/336934/raise-128kib-limit-on-environment-variables-in-linux )

Wanted solution

  • There should be a database entry containing the config as json.
  • The config shall be changeable by rest call (Usecase
  • The server will validate config and denies invalid ones by HTTP code NOT ACCEPTABLE
  • Because having no cluster wide events (at the moment) we do not use events, but a scheduler mechanism (or caching db based ?) to synchronize changes - using the version field from db entity should be enough to handle synch.
  • changes on scan config should inform admins
@de-jcup
Copy link
Member Author

de-jcup commented Feb 12, 2020

With a look at #2 we must handle this carefully and consider configuration in big picture.

When having a web UI (#11 ) we need REST operations able to to handle actions like in next mockup:
image

Because we create an own backend application for #11 we are not forced to handle each operation as necessary for mock ui - but it's okay to make REST calls not too different/difficult...

Domain: Administration

  • SUPER_ADMIN must be able to configure by REST
  • https://sechubserver/api/admin/config/mapping/$mappingId/ (PUT/GET:JSON)
    so we hide the adapter part here - mapping could be something not only for adapters.
  • JSON contains a list containing pattern,replacement,comment entries - with a look at the UI: JSON contains full table content (so we got ordering, delete automatically/implicit )
  • store in adapter configuration mapping entity/table (NEW)
  • for later adapter common configuration we use another table!
  • identifier for all these tables will be adapterID

Domain: Scan

  • can be kept as is (at least at the moment)
  • must store config inside database
  • initial config must be challenged - is this really the way to go? Do we need this after this
    issue? maybe we cut the initial part
  • config service must fetch data from DB and cache it some time
  • must provide event message handler (receive)
    • ADAPTER_CONFIG_CHANGED
    • SCANCONFIG_RECALCULATION

Shared kernel:

  • identifier for scan config entries must be shared
  • also which product supports which name pattern mappings!
  • UC_ADMIN_CONFIGURES_MAPPING

@de-jcup de-jcup self-assigned this Feb 12, 2020
@de-jcup
Copy link
Member Author

de-jcup commented Feb 12, 2020

Developer admin UI

To provide this as simple as possible inside developer admin UI, we will provide

  • CSV Import (3 columns) -> will be transfered to json
  • CSV Export (3 columns) -> JSON to CSV
  • 2 Actions, shall be "Adapter->export/import configuration" -> combobox chooser (id)

de-jcup added a commit that referenced this issue Feb 12, 2020
- providing db entities, validation, service and rest
- added integration tests
@de-jcup
Copy link
Member Author

de-jcup commented Feb 12, 2020

We let administrators add also additional mappings (custom identifiers) no restrictions at the moment.
A delete will also not be implemented this time. If somebody has accidently added a new mapping it can be reset to an empty value (means empty mapping data entity).

de-jcup added a commit that referenced this issue Feb 13, 2020
- added integration test: changes in administration
  domain trigger changes in scan domain as well and scan
  will use triggered mappings.
- integrated messaging for mappings
- updated coding conventions ("transactional" services)
- storing scan config into db
- scan domain does now have a periodical refresh check
  for scan configuration updates (so db synchronizes cluster)
- changed integration test parts
- dropped former initial config parts
- restdoc documenation added
de-jcup added a commit that referenced this issue Feb 13, 2020
de-jcup added a commit that referenced this issue Feb 14, 2020
- mapping identifier does now provide
  - self sanity check (duplicates, valid id)
  - mapping type
- scan domain does only accept global or adapter configuration
  mapping types now
de-jcup added a commit that referenced this issue Feb 14, 2020
- CSV import/export for mappings
- Try as ScanConfig added (needed dependency to sechub-scan inside
  sechub-developertools) so admin is able to test mapping results
@de-jcup
Copy link
Member Author

de-jcup commented Feb 14, 2020

From main menu calling "->Adapter->Checkmarx" an adapter dialog appears, where mappings for adapter can be customized at runtime:

image

@de-jcup de-jcup closed this as completed Feb 14, 2020
de-jcup added a commit that referenced this issue Feb 14, 2020
- providing db entities, validation, service and rest
- added integration tests
de-jcup added a commit that referenced this issue Feb 14, 2020
- added integration test: changes in administration
  domain trigger changes in scan domain as well and scan
  will use triggered mappings.
- integrated messaging for mappings
- updated coding conventions ("transactional" services)
- storing scan config into db
- scan domain does now have a periodical refresh check
  for scan configuration updates (so db synchronizes cluster)
- changed integration test parts
- dropped former initial config parts
- restdoc documenation added
de-jcup added a commit that referenced this issue Feb 14, 2020
de-jcup added a commit that referenced this issue Feb 14, 2020
- mapping identifier does now provide
  - self sanity check (duplicates, valid id)
  - mapping type
- scan domain does only accept global or adapter configuration
  mapping types now
de-jcup added a commit that referenced this issue Feb 14, 2020
- CSV import/export for mappings
- Try as ScanConfig added (needed dependency to sechub-scan inside
  sechub-developertools) so admin is able to test mapping results
de-jcup added a commit that referenced this issue Feb 17, 2020
de-jcup added a commit that referenced this issue Feb 17, 2020
- Dropped integration test rest controller method for direct
  refresh of scan config (we just wait for periodic update!)
- integration test does now check every second for scan config refresh
- switched log statement to trace when same configuration detected
- changed integration tests so no waiting for scan config refresh
- added missing service annotation
@de-jcup de-jcup changed the title Provide ScanConfig setup changes without restarts Mapping configuration concept Feb 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant