feat(couchdb-java-client): Migrating from ektorp to cloudant #1149

smrutis1 · 2021-03-17T09:29:36Z

Please provide a summary of your changes here.

Which issue is this pull request belonging to and how is it solving it? (Migrating from Ektorp to Cloudant Java Client #1148)

Did you add or update any new dependencies that are required for your change? - Yes

Have added a new library(cloudant java client) instead of ektorp for DB operations.

Issue: Closes #1148

Suggest Reviewer

You can suggest reviewers here with an @mention.

How To Test?

Need to

Have you implemented any additional tests? - No

Checklist

Must:

All related issues are referenced in commit messages and in PR

Signed-off-by: Smruti Prakash Sahoo smruti.sahoo@siemens.com

mcjaeger · 2021-04-15T21:41:15Z

test run cleanly on vagrant (Java 11 based, Ubuntu 20, Couchdb 311)

JaideepPalit

Reviewed the PR.
Kindly have a look at the review comments.

JaideepPalit · 2021-04-22T05:15:02Z

...end/src-common/src/main/java/org/eclipse/sw360/datahandler/db/AttachmentDatabaseHandler.java

-        List<DocumentOperationResult> results = attachmentUsageRepository.executeBulk(sanitizedUsages);
-        if (!results.isEmpty()) {
+        List<Response> results = attachmentUsageRepository.executeBulk(sanitizedUsages);
+        if (CommonUtils.isNotEmpty(sanitizedUsages) && results.isEmpty()) {
            throw new SW360Exception("Some of the usage documents could not be created: " + results);


results seems to be empty , maybe it would be good to change exception message as well since previously exception was thrown when not empty
same for line 201

Have validated the response/results before exception

JaideepPalit · 2021-04-22T05:31:57Z

...end/src-common/src/main/java/org/eclipse/sw360/datahandler/db/AttachmentDatabaseHandler.java

@@ -185,20 +184,20 @@ public AttachmentUsage updateAttachmentUsage(AttachmentUsage attachmentUsage) {
    }

    public void updateAttachmentUsages(List<AttachmentUsage> attachmentUsages) throws TException {
-        List<DocumentOperationResult> results = attachmentUsageRepository.executeBulk(attachmentUsages);
+        List<Response> results = attachmentUsageRepository.executeBulk(attachmentUsages);


From the documentation, it specifies, that for each record/request there is a response and that response should be validated with statusCode and getError() to ensure no failed records before the next check in line 188-190.

Based on java docs from com.cloudant.client.api.database.class

* * Note that the value returned by {@code getStatusCode()} on each {@code Response} object is * the overall status returned from {@code bulk_docs} and will therefore be the same for all * {@code Response} objects. * * * The returned list of {@code Response}s should be examined to ensure that all of the documents submitted * in the original request were successfully added to the database. * * * When a document (or document revision) is not correctly committed to the database because of * an error, you must check the value of {@code getError()} to determine error type and course * of action. See * <a href="https://console.bluemix.net/docs/services/Cloudant/api/document.html#bulk-document-validation-and-conflict-errors" target="_blank"> * Bulk Document Validation and Conflict Errors</a> for more information. *

Have added the validation

JaideepPalit · 2021-04-22T07:02:02Z

...mon/src/main/java/org/eclipse/sw360/datahandler/db/ProjectVulnerabilityRatingRepository.java

    }

    @View(name = "projectVulnerabilityRating", map = PROJECT_VULNERABILITY_LINK)
    public List<ProjectVulnerabilityRating> getProjectVulnerabilityRating(String projectId) {
        List<ProjectVulnerabilityRating> queryResults = queryByPrefix("projectVulnerabilityRating", projectId);

        if (queryResults.size() > 1) {
-            log.error("More than one projectVulnerabilityRating found for project with id " + projectId);
+//            log.error("More than one projectVulnerabilityRating found for project with id " + projectId);


Remove/uncomment commented line of code

Have added the logger

JaideepPalit · 2021-04-22T07:15:31Z

...end/src/src-licenses/src/main/java/org/eclipse/sw360/licenses/db/LicenseDatabaseHandler.java

@@ -478,7 +482,7 @@ public RequestStatus updateLicenseFromAdditionsAndDeletions(License licenseAddit
        if (!PermissionUtils.isUserAtLeast(UserGroup.CLEARING_ADMIN, user)){
            return null;
        }
-        final List<DocumentOperationResult> documentOperationResults = licenseTypeRepository.executeBulk(licenseTypes);
+        final List<Response> documentOperationResults = licenseTypeRepository.executeBulk(licenseTypes);


Kindly check whether returned Response needs to be validated for any error

Have added the validation

backend/src/src-licenses/src/test/java/org/eclipse/sw360/licenses/LicenseHandlerTest.java

...datahandler/src/main/java/org/eclipse/sw360/datahandler/common/CustomThriftDeserializer.java

...ies/lib-datahandler/src/main/java/org/eclipse/sw360/datahandler/common/DatabaseSettings.java

...lib-datahandler/src/main/java/org/eclipse/sw360/datahandler/couchdb/AttachmentConnector.java

...lib-datahandler/src/test/java/org/eclipse/sw360/datahandler/common/DatabaseSettingsTest.java

...ndler/src/test/java/org/eclipse/sw360/datahandler/couchdb/AttachmentStreamConnectorTest.java

ravi110336 · 2021-04-22T08:59:37Z

Tested the UI and it is working fine for most of the cases but found some below issue, Please check:

Issue 1:
Add release to a component and while fetching the release details getting the error message " Error:Error fetching release from backend." ---> Working fine.

Issue 2:
Similar issue from Admin->DataBaseSanitation->Releases with more than one source attachment:
getting the error message " Error:Error fetching release from backend." --->Working fine.

Issue 3:
ECC-> click on release. getting the error message " Error:Error fetching release from backend." --->Working fine.

Observation 1: (Not sure if this is an issue )
Go to Requests -> click on any request -> blank result no output. ---> Working fine.

smrutis1 · 2021-04-23T05:55:34Z

@JaideepPalit , Thanks for the review. Have incorporated most of them. Kindly have a look and let me know. @ravi110336 Thanks for the test. Have fixed the mentioned issues however the observation regarding "Request" tab, I am not able to reproduce it.

smrutis1 · 2021-04-30T07:12:23Z

@ravi110336 , From discussion, have rectified the issue with moderation request. Kindly have a look

ravi110336 · 2021-04-30T07:41:43Z

Tested the moderation request and attachment download working fine.

mcjaeger · 2021-04-30T08:21:16Z

tested also with vagrant: looks good

JaideepPalit

Re-reviewed the PR.
Kindly have a look at some very minor comments.

Note-
The travis-ci logs gives lot of error messages during test, which are not coming in other branches.
https://travis-ci.org/github/eclipse/sw360/jobs/768972323#L2046

Apr 30, 2021 6:56:28 AM com.cloudant.client.org.lightcouch.CouchDbClient createDB
INFO: Created Database: 'sw360_test_db'
2021-04-30 06:56:28 ERROR DatabaseConnectorCloudant:138 - Error fetching document of type DesignDocument with id _design/AttachmentUsage : 404 Object Not Found at http://localhost:5984/sw360_test_db/_design/AttachmentUsage. Error: not_found. Reason: missing.
2021-04-30 06:56:28 ERROR DatabaseConnectorCloudant:138 - Error fetching document of type DesignDocument with id _design/Attachment : 404 Object Not Found at http://localhost:5984/sw360_test_db/_design/Attachment. Error: not_found. Reason: missing.
2021-04-30 06:56:28 ERROR DatabaseConnectorCloudant:138 - Error fetching document of type DesignDocument with id _design/Source : 404 Object Not Found at http://localhost:5984/sw360_test_db/_design/Source. Error: not_found. Reason: missing.
2021-04-30 06:56:28 ERROR DatabaseConnectorCloudant:138 - Error fetching document of type DesignDocument with id _design/Vendor : 404 Object Not Found at http://localhost:5984/sw360_test_db/_design/Vendor. Error: not_found. Reason: missing.
2021-04-30 06:56:28 ERROR DatabaseConnectorCloudant:138 - Error fetching document of type DesignDocument with id _design/Release : 404 Object Not Found at http://localhost:5984/sw360_test_db/_design/Release. Error: not_found. Reason: missing.
2021-04-30 06:56:29 ERROR DatabaseConnectorCloudant:138 - Error fetching document of type DesignDocument with id _design/Component : 404 Object Not Found at http://localhost:5984/sw360_test_db/_design/Component. Error: not_found. Reason: missing.
2021-04-30 06:56:29 ERROR DatabaseConnectorCloudant:138 - Error fetching document of type DesignDocument with id _design/Project : 404 Object Not Found at http://localhost:5984/sw360_test_db/_design/Project. Error: not_found. Reason: missing.
2021-04-30 06:56:29 ERROR DatabaseConnectorCloudant:138 - Error fetching document of type DesignDocument with id _design/User : 404 Object Not Found at http://localhost:5984/sw360_test_db/_design/User. Error: not_found. Reason: missing.
2021-04-30 06:56:29 ERROR DatabaseConnectorCloudant:138 - Error fetching document of type DesignDocument with id _design/ProjectVulnerabilityRating : 404 Object Not Found at http://localhost:5984/sw360_test_db/_design/ProjectVulnerabilityRating. Error: not_found. Reason: missing.
2021-04-30 06:56:29 ERROR DatabaseConnectorCloudant:138 - Error fetching document of type DesignDocument with id

Seems like , using cloudant - design docs are not getting generated.
It should also be tested on a clean system with no DBs or Design docs

libraries/lib-datahandler/pom.xml

...er/src/main/java/org/eclipse/sw360/datahandler/cloudantclient/DatabaseConnectorCloudant.java

...main/java/org/eclipse/sw360/datahandler/cloudantclient/DatabaseRepositoryCloudantClient.java

...src-cvesearch/src/main/java/org/eclipse/sw360/cvesearch/datasink/VulnerabilityConnector.java

backend/src/src-components/src/test/java/org/eclipse/sw360/components/ComponentHandlerTest.java

backend/src/src-changelogs/src/main/java/org/eclipse/sw360/changelogs/ChangeLogsHandler.java

ravi110336 · 2021-04-30T13:44:21Z

I have tested the branch after deleting all the DBs and design docs in my local system, all the DBs are getting created with the design docs successfully.

Signed-off-by: Smruti Prakash Sahoo <smruti.sahoo@siemens.com>

smrutis1 · 2021-05-06T09:25:21Z

@JaideepPalit Thanks for review. Have incorporated changes.

Note-
The travis-ci logs gives lot of error messages during test, which are not coming in other branches.
https://travis-ci.org/github/eclipse/sw360/jobs/768972323#L2046

(from discussion): This is due to added logger message, it's ok to keep it.

Have added the paginated view for components to mitigate loading issue

JaideepPalit

Reviewed the PR.
Kindly have a look at some review comments.
Also seems to be some issue with

sorting based on vendor and main licenses
- case-sensitivity
- Multivalue list
There seems to some issue with pagination. Based on columns sorted like vendor/license/name - total result seems to vary

...360-portlet/src/main/java/org/eclipse/sw360/portal/portlets/components/ComponentPortlet.java

JaideepPalit · 2021-05-06T09:56:14Z

backend/src-common/src/main/java/org/eclipse/sw360/datahandler/db/ComponentRepository.java

+        case 3:
+            query = getConnector().createQuery(Component.class, "bycomponenttype");
+            break;
+


Fix indentation for switch - case
extra newline could also be removed

Removed new line. Tried with CTRL+SHIFT+F, didn't help with switch case indentation.

backend/src-common/src/main/java/org/eclipse/sw360/datahandler/db/ComponentRepository.java

Signed-off-by: Smruti Prakash Sahoo <smruti.sahoo@siemens.com>

smrutis1 · 2021-05-07T12:42:23Z

Reviewed the PR.
Kindly have a look at some review comments.
Also seems to be some issue with

sorting based on vendor and main licenses

case-sensitivity

Multivalue list\

(from discussion) Raised an issue #1199

There seems to some issue with pagination. Based on columns sorted like vendor/license/name - total result seems to vary

Fixed it.

JaideepPalit

Code looks fine to me.

mcjaeger · 2021-05-07T13:07:08Z

has been tested on our staging server, vagrant of course and again staging server....

smrutis1 force-pushed the feat/CloudantMigrate branch from 7b03835 to ff7dbd4 Compare March 17, 2021 12:05

smrutis1 added the WIP work in progress label Mar 18, 2021

mcjaeger mentioned this pull request Mar 24, 2021

improve CouchDB performance by implementing "Joins" on repository level #125

Closed

smrutis1 force-pushed the feat/CloudantMigrate branch from ff7dbd4 to 8be1574 Compare April 14, 2021 06:46

smrutis1 force-pushed the feat/CloudantMigrate branch from 8be1574 to 7f935fe Compare April 20, 2021 06:36

smrutis1 added needs code review needs general test This is general testing, meaning that there is no org specific issue to check for and removed WIP work in progress labels Apr 20, 2021

smrutis1 force-pushed the feat/CloudantMigrate branch from 7f935fe to 5b43433 Compare April 21, 2021 13:28

JaideepPalit suggested changes Apr 22, 2021

View reviewed changes

JaideepPalit added needs rework This doesn't seem right after review and removed needs code review labels Apr 22, 2021

smrutis1 force-pushed the feat/CloudantMigrate branch 3 times, most recently from 36f301e to b4d3d90 Compare April 23, 2021 05:33

smrutis1 added needs code review and removed needs rework This doesn't seem right after review labels Apr 23, 2021

smrutis1 force-pushed the feat/CloudantMigrate branch 3 times, most recently from cf7b624 to 466a359 Compare April 30, 2021 06:48

smrutis1 requested a review from JaideepPalit April 30, 2021 07:10

JaideepPalit suggested changes Apr 30, 2021

View reviewed changes

JaideepPalit removed needs code review needs general test This is general testing, meaning that there is no org specific issue to check for labels Apr 30, 2021

JaideepPalit added the needs rework This doesn't seem right after review label Apr 30, 2021

feat(cloudant): Migrating from ektorp to cloudant java client

fd95a2c

Signed-off-by: Smruti Prakash Sahoo <smruti.sahoo@siemens.com>

smrutis1 force-pushed the feat/CloudantMigrate branch from 466a359 to ce72095 Compare May 6, 2021 08:52

smrutis1 added needs code review needs general test This is general testing, meaning that there is no org specific issue to check for and removed needs rework This doesn't seem right after review labels May 6, 2021

JaideepPalit suggested changes May 7, 2021

View reviewed changes

JaideepPalit added needs rework This doesn't seem right after review and removed needs code review needs general test This is general testing, meaning that there is no org specific issue to check for labels May 7, 2021

smrutis1 force-pushed the feat/CloudantMigrate branch from ce72095 to 0f5adba Compare May 7, 2021 12:06

feat(components-pagination): paginated view response for components

0e22d55

Signed-off-by: Smruti Prakash Sahoo <smruti.sahoo@siemens.com>

smrutis1 force-pushed the feat/CloudantMigrate branch from 0f5adba to 0e22d55 Compare May 7, 2021 12:18

smrutis1 mentioned this pull request May 7, 2021

Issue with sorting table on components tab #1199

Open

smrutis1 added needs code review and removed needs rework This doesn't seem right after review labels May 7, 2021

JaideepPalit approved these changes May 7, 2021

View reviewed changes

mcjaeger merged commit a91c7a9 into eclipse-sw360:master May 7, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(couchdb-java-client): Migrating from ektorp to cloudant #1149

feat(couchdb-java-client): Migrating from ektorp to cloudant #1149

smrutis1 commented Mar 17, 2021

mcjaeger commented Apr 15, 2021

JaideepPalit left a comment

JaideepPalit Apr 22, 2021

smrutis1 Apr 23, 2021

JaideepPalit Apr 22, 2021

smrutis1 Apr 23, 2021

JaideepPalit Apr 22, 2021

smrutis1 Apr 23, 2021

JaideepPalit Apr 22, 2021

smrutis1 Apr 23, 2021

ravi110336 commented Apr 22, 2021 •

edited

Loading

smrutis1 commented Apr 23, 2021

smrutis1 commented Apr 30, 2021

ravi110336 commented Apr 30, 2021

mcjaeger commented Apr 30, 2021

JaideepPalit left a comment

ravi110336 commented Apr 30, 2021

smrutis1 commented May 6, 2021

JaideepPalit left a comment

JaideepPalit May 6, 2021

smrutis1 May 7, 2021 •

edited

Loading

smrutis1 commented May 7, 2021

JaideepPalit left a comment

mcjaeger commented May 7, 2021

feat(couchdb-java-client): Migrating from ektorp to cloudant #1149

feat(couchdb-java-client): Migrating from ektorp to cloudant #1149

Conversation

smrutis1 commented Mar 17, 2021

Suggest Reviewer

How To Test?

Checklist

mcjaeger commented Apr 15, 2021

JaideepPalit left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ravi110336 commented Apr 22, 2021 • edited Loading

smrutis1 commented Apr 23, 2021

smrutis1 commented Apr 30, 2021

ravi110336 commented Apr 30, 2021

mcjaeger commented Apr 30, 2021

JaideepPalit left a comment

Choose a reason for hiding this comment

ravi110336 commented Apr 30, 2021

smrutis1 commented May 6, 2021

JaideepPalit left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smrutis1 May 7, 2021 • edited Loading

Choose a reason for hiding this comment

smrutis1 commented May 7, 2021

JaideepPalit left a comment

Choose a reason for hiding this comment

mcjaeger commented May 7, 2021

ravi110336 commented Apr 22, 2021 •

edited

Loading

smrutis1 May 7, 2021 •

edited

Loading