Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ca 294 refactor library management #321

Merged
merged 96 commits into from
May 16, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
96 commits
Select commit Hold shift + click to select a range
a5e0aba
Horrible initial commit for library management
monsieurswag Apr 24, 2024
e4a7be7
Resolve initial conflicts
monsieurswag Apr 24, 2024
cb94245
Fix library detail view basic objects display
monsieurswag Apr 24, 2024
d846d97
The dependencies field is not a JSONField
monsieurswag Apr 24, 2024
762b660
Remove useless special serializing method from the StoredLibraryDetai…
monsieurswag Apr 24, 2024
a00d456
Fix typo in the AbstractBaseModel
monsieurswag Apr 24, 2024
fe291db
Fix double API urlmodel related bugs
monsieurswag Apr 27, 2024
6e30549
Remove test code
nas-tabchiche Apr 29, 2024
3650a68
Merge branch 'main' into CA-294-Refactor-library-management
nas-tabchiche Apr 29, 2024
c3ff617
chore: Merge conflicting migrations
nas-tabchiche Apr 29, 2024
e8ac2a3
chore: Run formatter
nas-tabchiche Apr 29, 2024
a329fc6
Fix some typing and typo
nas-tabchiche Apr 29, 2024
a52a555
Undo regression-inducing changes to ModelTable component
nas-tabchiche Apr 29, 2024
0e4dd16
Improve libraries table row actions management
nas-tabchiche Apr 29, 2024
aaac73b
Move stored library initialization in app.startup
nas-tabchiche Apr 29, 2024
af4f9b3
chore: Run formatter
nas-tabchiche Apr 29, 2024
0c8d739
WIP: API tests
nas-tabchiche Apr 29, 2024
bc0a49d
Initialize stored libraries in a management command
nas-tabchiche Apr 30, 2024
50cb4f2
Refactor StoredLibrary model and LibraryMixin
nas-tabchiche Apr 30, 2024
09eb41e
Fix typo: libary -> library
nas-tabchiche Apr 30, 2024
45b2205
Subclass BaseModelViewset for StoredLibraryViewSet
nas-tabchiche Apr 30, 2024
933b447
Remove dead code
nas-tabchiche Apr 30, 2024
9c036ff
Refactor library content and tree display
nas-tabchiche Apr 30, 2024
9186795
Add useFocusTrap prop
nas-tabchiche Apr 30, 2024
0820115
Stop using focus traps for library upload form
nas-tabchiche Apr 30, 2024
9dca9e0
Delete debug logs
nas-tabchiche Apr 30, 2024
2aed24e
Update storelibraries command
nas-tabchiche Apr 30, 2024
7c1135b
Call storelibraries command from core.apps.startup
nas-tabchiche Apr 30, 2024
c342c25
chore: Run migrations
nas-tabchiche Apr 30, 2024
dcbba44
Merge branch 'main' into CA-294-Refactor-library-management
nas-tabchiche Apr 30, 2024
544244e
Merge branch 'main' into CA-294-Refactor-library-management
nas-tabchiche Apr 30, 2024
a2df2b4
chore: Run migrations
nas-tabchiche Apr 30, 2024
04791e7
Add null check for source.meta
nas-tabchiche May 1, 2024
3a2bf84
Rename StoredLibrary.loads -> load
nas-tabchiche May 1, 2024
1d6e062
Fix unit tests
nas-tabchiche May 1, 2024
bf862fc
chore: Run formatter
nas-tabchiche May 1, 2024
efcbdbe
Fix href in functional tests
nas-tabchiche May 1, 2024
53203df
Merge branch 'main' into CA-294-Refactor-library-management
nas-tabchiche May 2, 2024
33006f2
Pass API tests (finally)
nas-tabchiche May 3, 2024
61bccbc
Fix ruff pre-commit hook
nas-tabchiche May 3, 2024
a9d6f40
Disable ruff linting in pre-commit
nas-tabchiche May 3, 2024
a2c1494
chore: Run formatter
nas-tabchiche May 3, 2024
c55b389
Merge branch 'main' into CA-294-Refactor-library-management
nas-tabchiche May 3, 2024
4d6918b
pre-commit: Exclude migrations files from format checking
nas-tabchiche May 3, 2024
0b6b76a
chore: Run ruff format
nas-tabchiche May 3, 2024
3f87144
chore: Run migrations
nas-tabchiche May 3, 2024
6a0c87a
Fix stack trace information exposure on library storage failure
nas-tabchiche May 3, 2024
53d507b
Fix default django host in e2e tests script
nas-tabchiche May 3, 2024
24937dc
Pass library import functional test
nas-tabchiche May 3, 2024
4ae651c
Fix language tag functional test
nas-tabchiche May 3, 2024
586aea3
Fix library import util
nas-tabchiche May 3, 2024
4966989
Merge branch 'main' into CA-294-Refactor-library-management
nas-tabchiche May 10, 2024
a84e3b0
Fix typing in crud.ts
nas-tabchiche May 12, 2024
f779700
Remove 'search' attribute from model definitions
nas-tabchiche May 12, 2024
876ea8d
ModelTable: Fix preventDelete login
nas-tabchiche May 12, 2024
0a1a70a
Merge branch 'main' into CA-294-Refactor-library-management
nas-tabchiche May 12, 2024
c385810
Fix typing in crud.ts
nas-tabchiche May 12, 2024
4d21dc7
Use matrix strategy for functional tests workflow
nas-tabchiche May 12, 2024
2cbdc3e
Playwright: Install only needed browsers
nas-tabchiche May 12, 2024
558f506
Update artifact upload step on functional tests workflow
nas-tabchiche May 12, 2024
abcfec9
Remove debug prints
nas-tabchiche May 12, 2024
db4036e
chore: Remove dead code
nas-tabchiche May 12, 2024
952f58e
Merge branch 'main' into CA-294-Refactor-library-management
eric-intuitem May 13, 2024
cd155ed
Add the builtin attribute to all builtin library files
monsieurswag May 14, 2024
f274849
Add a tool to add the builtin attribute to all builtin libary file wh…
monsieurswag May 14, 2024
821f94f
Replace the term imported library by loaded library in translations f…
monsieurswag May 14, 2024
f8783a3
Rename the is_imported field of the StoredLibrary model by is_loaded
monsieurswag May 14, 2024
5559c14
Change the library loading success message to remove the word importe…
monsieurswag May 14, 2024
4c9f4e3
Adapt unit tests and functional tests to the library translation changes
monsieurswag May 14, 2024
680dee3
Make the custom stored libraries deletable
monsieurswag May 14, 2024
4b9c62a
Replace the term upload library to add library in translation files
monsieurswag May 14, 2024
c02b536
Fix non-clickable libraries link in breadcrumb
monsieurswag May 14, 2024
61a5b20
Improve library delete message
monsieurswag May 14, 2024
c5ad388
Fix migration issue by renaming library to loadedlibrary
monsieurswag May 14, 2024
84578be
Fix the delete button not showing for loaded builtin libarries
monsieurswag May 14, 2024
0cc1ae7
Formatter
ab-smith May 14, 2024
debaa8f
Set the is_loaded attribute at the initialization of stored libraries
monsieurswag May 15, 2024
42731aa
Set the objects_meta attribute at the migration of the library model …
monsieurswag May 15, 2024
fa6e99c
Display stored libraries that have been loaded when they are not builtin
monsieurswag May 15, 2024
08d0ba5
Fix failing detail views
monsieurswag May 15, 2024
27b3e93
Formatter
monsieurswag May 15, 2024
1381c98
Prettier
monsieurswag May 15, 2024
466e466
Fix backend unit tests
monsieurswag May 15, 2024
5ae1662
Formatter
monsieurswag May 15, 2024
f3e56f5
Update data-model
monsieurswag May 15, 2024
0cac3a3
Add the deutsch language flag display
monsieurswag May 15, 2024
d807671
Keep builtin library in the library store after loading them
monsieurswag May 15, 2024
9a13447
Remove the is_obsolete attribute from the StoredLibrary model
monsieurswag May 15, 2024
7af9afb
Make the StoredLibrary builtin attribute non-customizable
monsieurswag May 15, 2024
7a311f8
Formatter
monsieurswag May 15, 2024
091ffee
fix builtin
eric-intuitem May 15, 2024
2def62a
fix number shown for framework object
eric-intuitem May 15, 2024
ea4d3ee
feat: show import button for library stored detailed view
Mohamed-Hacene May 16, 2024
7566eff
ci/cd: fix functional library import
Mohamed-Hacene May 16, 2024
a41b0a3
Fix functional test for library import
nas-tabchiche May 16, 2024
a02f02a
Merge branch 'main' into CA-294-Refactor-library-management
nas-tabchiche May 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
18 changes: 8 additions & 10 deletions .github/workflows/functional-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,15 +27,13 @@ jobs:
options: --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
mailhog:
image: mailhog/mailhog
ports: [
"1025:1025",
"8025:8025"
]
ports: ["1025:1025", "8025:8025"]

strategy:
max-parallel: 4
matrix:
python-version: ["3.11"]
playwright-browser: ["chromium", "firefox"]

steps:
- uses: actions/checkout@v4
Expand All @@ -55,9 +53,9 @@ jobs:
- name: Install dependencies
working-directory: ${{ env.working-directory }}
run: npm ci
- name: Install Playwright Browsers
- name: Install Playwright browser ${{ matrix.playwright-browser }}
working-directory: ${{ env.working-directory }}
run: npx playwright install --with-deps
run: npx playwright install --with-deps ${{ matrix.playwright-browser }}
- name: Create frontend environment variables file
working-directory: ${{ env.working-directory }}
run: |
Expand Down Expand Up @@ -93,14 +91,14 @@ jobs:
export $(grep -v '^#' .env | xargs)
python manage.py createsuperuser --noinput
nohup python manage.py runserver &
- name: Run tests
- name: Run tests with browser ${{ matrix.playwright-browser }}
working-directory: ${{ env.working-directory }}
run: npx playwright test
run: npx playwright test --project=${{ matrix.playwright-browser }}
- uses: actions/upload-artifact@v4
if: always()
with:
name: functional-tests-report
path: |
${{ env.working-directory }}/tests/results/
${{ env.working-directory }}/tests/reports/
${{ env.working-directory }}/tests/${{ matrix.playwright-browser }}results/
${{ env.working-directory }}/tests/${{ matrix.playwright-browser }}reports/
retention-days: 5
5 changes: 3 additions & 2 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,11 +9,12 @@ repos:
rev: v0.4.1
hooks:
# Run the linter.
- id: ruff
args: [--fix backend]
# - id: ruff
# args: [--fix, backend]
# Run the formatter.
- id: ruff-format
args: [backend]
exclude: (migrations)
- repo: local
hooks:
- id: format-frontend
Expand Down
48 changes: 31 additions & 17 deletions backend/app_tests/api/test_api_libraries.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
import json
import pytest
from rest_framework.test import APIClient
from core.models import Framework
from app_tests.test_vars import TEST_FRAMEWORK_URN, TEST_RISK_MATRIX_URN
from core.models import Framework, StoredLibrary
from core.models import RiskMatrix
from iam.models import Folder
from rest_framework import status
Expand All @@ -17,7 +19,7 @@ class TestLibrariesUnauthenticated:
def test_get_libraries(self):
"""test to get libraries from the API without authentication"""

EndpointTestsQueries.get_object(self.client, "Libraries")
EndpointTestsQueries.get_object(self.client, "Stored libraries")

def test_import_frameworks(self):
"""test to import libraries with the API without authentication"""
Expand Down Expand Up @@ -50,21 +52,27 @@ def test_get_libraries(self, test):
"""test to get libraries from the API with authentication"""

EndpointTestsQueries.Auth.get_object(
test.client, "Libraries", base_count=-1, user_group=test.user_group
test.client, "Stored libraries", base_count=-1, user_group=test.user_group
)

def test_import_frameworks(self, test):
"""test to import frameworks with the API with authentication"""

# Uses the API endpoint to get library details with the admin client
lib_detail_response = test.admin_client.get(
EndpointTestsUtils.get_object_urn("Framework")
).json()["objects"]["framework"]
EndpointTestsUtils.get_stored_library_content(
test.client, TEST_FRAMEWORK_URN
)
)
lib_detail_response = lib_detail_response.content
lib_detail_response = json.loads(lib_detail_response)
lib_detail_response = json.loads(lib_detail_response)
lib_detail_response = lib_detail_response["framework"]

# Asserts that the library is not already imported
# Asserts that the library is not already loaded
assert (
Framework.objects.all().count() == 0
), "libraries are already imported in the database"
), "libraries are already loaded in the database"
EndpointTestsQueries.Auth.get_object(
test.client,
"Frameworks",
Expand All @@ -78,12 +86,12 @@ def test_import_frameworks(self, test):
assert Framework.objects.all().count() == (
1
if not EndpointTestsUtils.expected_request_response(
"add", "library", str(test.folder), test.user_group
"add", "loadedlibrary", str(test.folder), test.user_group
)[0]
else 0
), "Frameworks are not correctly imported in the database"

# Uses the API endpoint to assert that the library was properly imported
# Uses the API endpoint to assert that the library was properly loaded
EndpointTestsQueries.Auth.get_object(
test.client,
"Frameworks",
Expand All @@ -96,7 +104,7 @@ def test_import_frameworks(self, test):
base_count=1,
user_group=test.user_group,
fails=EndpointTestsUtils.expected_request_response(
"add", "library", str(test.folder), test.user_group
"add", "loadedlibrary", str(test.folder), test.user_group
)[0],
)

Expand All @@ -121,13 +129,19 @@ def test_import_risk_matrix(self, test):

# Uses the API endpoint to get library details with the admin client
lib_detail_response = test.admin_client.get(
EndpointTestsUtils.get_object_urn("Risk matrix")
).json()["objects"]["risk_matrix"][0]
EndpointTestsUtils.get_stored_library_content(
test.client, TEST_RISK_MATRIX_URN
)
)
lib_detail_response = lib_detail_response.content
lib_detail_response = json.loads(lib_detail_response)
lib_detail_response = json.loads(lib_detail_response)
lib_detail_response = lib_detail_response["risk_matrix"][0]

# Asserts that the library is not already imported
# Asserts that the library is not already loaded
assert (
RiskMatrix.objects.all().count() == 0
), "libraries are already imported in the database"
), "libraries are already loaded in the database"
EndpointTestsQueries.Auth.get_object(
test.client, "Risk matrices", user_group=test.user_group
)
Expand All @@ -139,12 +153,12 @@ def test_import_risk_matrix(self, test):
assert RiskMatrix.objects.all().count() == (
1
if not EndpointTestsUtils.expected_request_response(
"add", "library", str(test.folder), test.user_group
"add", "loadedlibrary", str(test.folder), test.user_group
)[0]
else 0
), "Risk matrices are not correctly imported in the database"

# Uses the API endpoint to assert that the library was properly imported
# Uses the API endpoint to assert that the library was properly loaded
EndpointTestsQueries.Auth.get_object(
test.client,
"Risk matrices",
Expand All @@ -158,7 +172,7 @@ def test_import_risk_matrix(self, test):
base_count=1,
user_group=test.user_group,
fails=EndpointTestsUtils.expected_request_response(
"add", "library", str(test.folder), test.user_group
"add", "loadedlibrary", str(test.folder), test.user_group
)[0],
)

Expand Down
7 changes: 5 additions & 2 deletions backend/app_tests/api/test_api_requirement_nodes.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import pytest
from rest_framework.test import APIClient
from core.models import RequirementNode, Framework
from app_tests.test_vars import TEST_FRAMEWORK_URN
from core.models import RequirementNode, Framework, StoredLibrary
from iam.models import Folder

from test_utils import EndpointTestsQueries, EndpointTestsUtils
Expand Down Expand Up @@ -76,7 +77,9 @@ def test_import_requirement_nodes(self, test):
test.client,
"Requirement nodes",
EndpointTestsUtils.get_endpoint_url("Requirement nodes"),
EndpointTestsUtils.get_object_urn("Framework"),
EndpointTestsUtils.get_stored_library_content(
test.client, TEST_FRAMEWORK_URN
),
[
"name",
"description",
Expand Down
25 changes: 20 additions & 5 deletions backend/app_tests/api/test_utils.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
from django.db import models
from knox.auth import AuthToken
import pytest
import json
Expand All @@ -6,6 +7,7 @@
from django.urls import reverse
from rest_framework import status
from rest_framework.test import APIClient
from core.models import StoredLibrary

from test_vars import *

Expand All @@ -25,7 +27,17 @@ def get_object_urn(object_name: str, resolved: bool = True):

urn_varname = format_urn(object_name)
urn = get_var(urn_varname)
return f"{reverse(LIBRARIES_ENDPOINT)}{urn}/" if resolved else eval(urn)
return f"{reverse(STORED_LIBRARIES_ENDPOINT)}{urn}/" if resolved else eval(urn)

def get_referential_object_url_from_urn(
authenticated_client, urn: str, model: models.Model = StoredLibrary
):
"""Get the object URL from the URN"""
return f"{reverse(STORED_LIBRARIES_ENDPOINT)}{urn}/"

def get_stored_library_content(authenticated_client, urn: str) -> str:
"""Return an URL to fetch the content of a stored library"""
return f"{reverse(STORED_LIBRARIES_ENDPOINT)}{urn}/content/"

@pytest.mark.django_db
def get_test_client_and_folder(
Expand Down Expand Up @@ -958,7 +970,7 @@ def import_object(
user_perm_expected_status,
user_perm_reason,
) = EndpointTestsUtils.expected_request_response(
"add", "library", scope, user_group, expected_status
"add", "loadedlibrary", scope, user_group, expected_status
)

url = urn or EndpointTestsUtils.get_object_urn(verbose_name)
Expand Down Expand Up @@ -1014,9 +1026,12 @@ def compare_results(
reference.status_code == status.HTTP_200_OK
), "reference endpoint is not accessible"

for object in reference.json()["objects"]["framework"][
object_name.lower().replace(" ", "_")
][:count]:
content = json.loads(reference.content)
content = json.loads(content)

for object in content["framework"][object_name.lower().replace(" ", "_")][
:count
]:
comparelist = authenticated_client.get(compare_url)
compare = dict()
assert (
Expand Down
2 changes: 1 addition & 1 deletion backend/app_tests/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ def __init__(self, *args, **kwargs):

@pytest.fixture
def app_config():
startup()
startup(sender=None, **{})


@pytest.fixture
Expand Down
4 changes: 2 additions & 2 deletions backend/app_tests/test_vars.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
EVIDENCES_ENDPOINT = "evidences-list"
FOLDERS_ENDPOINT = "folders-list"
FRAMEWORKS_ENDPOINT = "frameworks-list"
LIBRARIES_ENDPOINT = "libraries-list"
STORED_LIBRARIES_ENDPOINT = "stored-libraries-list"
RISK_MATRICES_ENDPOINT = "risk-matrices-list"
PROJECTS_ENDPOINT = "projects-list"
REQUIREMENT_ASSESSMENTS_ENDPOINT = "requirement-assessments-list"
Expand Down Expand Up @@ -95,7 +95,7 @@ def get_var(varname: str) -> Any:

def get_singular_name(plural_name: str) -> str:
exceptions = {
"Libraries": "Library",
"Stored libraries": "Stored library",
"Risk matrices": "Risk matrix",
"Policies": "Policy",
}
Expand Down
23 changes: 0 additions & 23 deletions backend/cal/forms.py

This file was deleted.

2 changes: 1 addition & 1 deletion backend/ciso_assistant/settings.py
Original file line number Diff line number Diff line change
Expand Up @@ -280,7 +280,7 @@ def set_ciso_assistant_url(_, __, event_dict):

# SQLIte file can be changed, useful for tests
SQLITE_FILE = os.environ.get("SQLITE_FILE", BASE_DIR / "db/ciso-assistant.sqlite3")

LIBRARIES_PATH = library_path = BASE_DIR / "library/libraries"

if "POSTGRES_NAME" in os.environ:
DATABASES = {
Expand Down