Skip to content

Commit

Permalink
🔀 : merge pull request #699 from gaia-app/feature/module-refresh-for-…
Browse files Browse the repository at this point in the history
…manually-imported-modules

Feature/module refresh for manually imported modules
  • Loading branch information
cdubuisson committed May 21, 2022
2 parents 831c15e + da01f1b commit 57e49d3
Show file tree
Hide file tree
Showing 14 changed files with 308 additions and 28 deletions.
4 changes: 2 additions & 2 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -12,12 +12,12 @@ jobs:
# fetching all history (to help sonar computing PRs)
fetch-depth: 0
- name: Set up JDK 17
uses: actions/setup-java@v1
uses: actions/setup-java@v3
with:
distribution: 'temurin'
java-version: 17
- name: Run Maven tests
run: mvn org.jacoco:jacoco-maven-plugin:prepare-agent verify org.jacoco:jacoco-maven-plugin:report sonar:sonar -DexcludedGroups=e2e -P ci-tu
run: mvn --batch-mode org.jacoco:jacoco-maven-plugin:prepare-agent verify org.jacoco:jacoco-maven-plugin:report sonar:sonar -DexcludedGroups=e2e -P ci-tu
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
4 changes: 2 additions & 2 deletions .github/workflows/visual-non-regression-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,13 +19,13 @@ jobs:
# fetching all history (to help sonar computing PRs)
fetch-depth: 0
- name: Set up JDK 17
uses: actions/setup-java@v1
uses: actions/setup-java@v3
with:
distribution: 'temurin'
java-version: 17
- name: Percy Test
uses: percy/exec-action@v0.3.1
with:
command: "mvn test -Dgroups=e2e"
command: "mvn --batch-mode test -Dgroups=e2e"
env:
PERCY_TOKEN: ${{ secrets.PERCY_TOKEN }}
56 changes: 56 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,61 @@
# Changelog

<a name="2.3.0"></a>
## 2.3.0 (2022-05-08)

### Added

- ✨ : update registry details when module is saved [[e15dbcd](https://github.com/gaia-app/gaia/commit/e15dbcd37af99caf5ec64a08f136d0f14aec1474)]
- ✨ : compute registry details for Gitlab urls [[8436476](https://github.com/gaia-app/gaia/commit/8436476813599b12e8f42c679e8872704e7493ba)]
* ✨ : compute registry details for Github ssh urls ([c0c3153](https://github.com/gaia-app/gaia/commit/c0c3153940fbcb795d434e45595d2eb9822b7be8))
* ✨ : compute registry details for Github https urls ([1bfcecc](https://github.com/gaia-app/gaia/commit/1bfceccb1e0fe928e141362c90ef0e6237cf7c2b))

### Changed

- ⬆️ : bump kotlin to 1.6.21 [[d569337](https://github.com/gaia-app/gaia/commit/d5693373fba3a01cfa0a0770333003d491ad1918)]
- ⬆️ : bump jacoco-maven-plugin to 0.8.8 [[4573ad1](https://github.com/gaia-app/gaia/commit/4573ad1e6a69c12801b28c091bce2028dc37c8ea)]
- ⬆️ : bump setup-java action to v3 [[8347df6](https://github.com/gaia-app/gaia/commit/8347df637648b4d18bed9ab6135a6d08b6cb7ec8)]
- ⬆️ : bump kotlin.version to 1.6.0 [[0004c28](https://github.com/gaia-app/gaia/commit/0004c28b0480910b3131b59aa99b386c5057e594)]
- 🔧 : add com.sun.jndi.ldap module export [[808b16d](https://github.com/gaia-app/gaia/commit/808b16d7a795d0bbb6a3375b21071be6ee5667d7)]
- ⬆️ : upgrade to java 17 [[a091b2a](https://github.com/gaia-app/gaia/commit/a091b2a96287b49e6be16cf782884ce7836cc631)]
* 👷 : migrate to java 17 ([c0d4fed](https://github.com/gaia-app/gaia/commit/c0d4fed44a9d1caa6353d0634f15629394694cbd))
* ⬆️ : upgrade to java 17 ([619bf1e](https://github.com/gaia-app/gaia/commit/619bf1e92922be21782911a75dec39ea3f17b5f8))
- 🔧 : update default mongodb URL to use 27017 port [[46a3376](https://github.com/gaia-app/gaia/commit/46a33765f1f50bff160b333ac67437dd3066b0db)]
- ♻️ : remove redundant semicolons [[da4c98e](https://github.com/gaia-app/gaia/commit/da4c98e2e90a44d8084117566ed5ad11add8a51e)]
- ♻️ : remove redundant qualifiers [[be3273b](https://github.com/gaia-app/gaia/commit/be3273bf6f40b83e352005b33e314832f5e40139)]
* 🔥 : remove redundent modifiers ([8649046](https://github.com/gaia-app/gaia/commit/86490460f744eee521a05eacd071bfd9db523169))
- ♻️ : remove unnecessary non-null assertion [[0aeb6eb](https://github.com/gaia-app/gaia/commit/0aeb6eb896944f0aa8007ba0b49b295c7456bd24)]
- ♻️ : replace field injection with constructor [[56de233](https://github.com/gaia-app/gaia/commit/56de2339f208ee4e8b3f44b9dca57646afc8d497)]
- ♻️ : replace null checks with ifPresent() [[446cdd9](https://github.com/gaia-app/gaia/commit/446cdd978d81a560580e8c5cd92babfa766b11e3)]
- ♻️ : replace addAll with parametrized constructor call [[7352877](https://github.com/gaia-app/gaia/commit/73528770688d6b537aa20f59c010f3349d8ad7a7)]
- ♻️ : replace unchecked assignment [[a5ff22b](https://github.com/gaia-app/gaia/commit/a5ff22b53eaf76790141098dc92a9ef0134006d7)]
- 🎨 : remove unnecessary semicolon [[7ae64ea](https://github.com/gaia-app/gaia/commit/7ae64ea7fdfd79a7298038905bf2f9a7cd76e6e5)]
- ♻️ : stop using deprecated constructor [[65e4b36](https://github.com/gaia-app/gaia/commit/65e4b36d4b3068f10cade84d15ecb83dd47f9a05)]
- ♻️ : use redirectUri instead of redirectUriTemplate [[492778b](https://github.com/gaia-app/gaia/commit/492778be3ea9cd383bb9a05d237d64a2e033d3a1)]
- 🚚 : split configuration of oauth providers [[bcee58c](https://github.com/gaia-app/gaia/commit/bcee58c44a81d1eacdcdda2f632d02d9dadc0ddc)]

### Removed

- 🔥 : remove unused imports [[7ffa067](https://github.com/gaia-app/gaia/commit/7ffa067560be69f00877f46efc2281b4523bd538)]
- 🔥 : remove redundant throws clause [[164fdcd](https://github.com/gaia-app/gaia/commit/164fdcde3e49bad4864bac5c85e4cde010b661a7)]
- 🔥 : remove unnecessary imports [[a230bdd](https://github.com/gaia-app/gaia/commit/a230bdd48438d4a72320c241102deeb9c65f423b)]
- 🔥 : remove unused dockerDaemon settings [[929914c](https://github.com/gaia-app/gaia/commit/929914cf8c084d705e8e8c56922283d2dd6bf19e)]

### Fixed

- 🐛 : group parts of the regex together to make the intended operator precedence explicit [[b1aeaca](https://github.com/gaia-app/gaia/commit/b1aeaca6edfbfc86fa90ceda614437e59143c410)]

### Security

- 🔒 : auto create admin user on startup [[f4ba7e9](https://github.com/gaia-app/gaia/commit/f4ba7e9ad4a2ec9ec36fdda206161d681b8653a4)]

### Miscellaneous

- 👷 : set maven to batch-mode [[4d5b722](https://github.com/gaia-app/gaia/commit/4d5b7227a98e7f2173cb49227be08230f0a44dce)]
- 🙈 : add node_modules to .dockerignore [[262ce92](https://github.com/gaia-app/gaia/commit/262ce92962ccd1d1fa2706af3f82a729541437fe)]
- 📝 : add documentation links [[6dcb23a](https://github.com/gaia-app/gaia/commit/6dcb23a998eb2e449002757ffe2fcc942702f012)]


<a name="2.2.0"></a>
## 2.2.0 (2021-08-20)

Expand Down
7 changes: 4 additions & 3 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@

<groupId>io.gaia_app</groupId>
<artifactId>gaia</artifactId>
<version>2.3.0-SNAPSHOT</version>
<version>2.3.0</version>
<packaging>jar</packaging>

<name>gaia</name>
Expand Down Expand Up @@ -46,13 +46,14 @@
<cucumber-jvm.version>6.10.0</cucumber-jvm.version>
<jersey.version>2.27</jersey.version>
<junit-jupiter.version>5.5.2</junit-jupiter.version>
<kotlin.version>1.6.0</kotlin.version>
<kotlin.version>1.6.21</kotlin.version>
<kotlinx.version>1.5.1</kotlinx.version>
<testcontainers.version>1.15.2</testcontainers.version>
<node.version>v12.16.0</node.version>
<npm.version>6.13.4</npm.version>
<spring.vault.version>2.2.2.RELEASE</spring.vault.version>
<frontend-maven-plugin.version>1.11.0</frontend-maven-plugin.version>
<jacoco-maven-plugin.version>0.8.8</jacoco-maven-plugin.version>

<sonar.projectKey>gaia-app:gaia</sonar.projectKey>
<sonar.organization>gaia-app</sonar.organization>
Expand Down Expand Up @@ -236,7 +237,7 @@
<plugin>
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.8.7</version>
<version>${jacoco-maven-plugin.version}</version>
</plugin>
</plugins>
</pluginManagement>
Expand Down
7 changes: 2 additions & 5 deletions src/main/java/io/gaia_app/credentials/Credentials.kt
Original file line number Diff line number Diff line change
@@ -1,17 +1,14 @@
package io.gaia_app.credentials


import com.fasterxml.jackson.annotation.JsonSubTypes
import com.fasterxml.jackson.annotation.JsonSubTypes.Type
import com.fasterxml.jackson.annotation.JsonTypeInfo
import org.springframework.data.annotation.Id


import com.fasterxml.jackson.annotation.JsonTypeInfo.As.PROPERTY
import com.fasterxml.jackson.annotation.JsonTypeInfo.Id.NAME
import com.fasterxml.jackson.annotation.JsonView
import com.fasterxml.jackson.databind.annotation.JsonSerialize
import io.gaia_app.teams.User
import org.springframework.data.annotation.CreatedBy
import org.springframework.data.annotation.Id
import org.springframework.data.mongodb.core.mapping.DBRef
import org.springframework.data.mongodb.core.mapping.Document

Expand Down
14 changes: 14 additions & 0 deletions src/main/java/io/gaia_app/modules/ModuleService.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
package io.gaia_app.modules;

import io.gaia_app.modules.bo.TerraformModule;

public interface ModuleService {

/**
* Updates the registry details for a module.
* It tries to compute a project id from an URL, and updates the module.
* @param module module to update
*/
void updateRegistryDetails(TerraformModule module);

}
81 changes: 81 additions & 0 deletions src/main/java/io/gaia_app/modules/ModuleServiceImpl.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
package io.gaia_app.modules;

import io.gaia_app.modules.bo.TerraformModule;
import io.gaia_app.registries.RegistryApi;
import io.gaia_app.registries.RegistryDetails;
import io.gaia_app.registries.RegistryType;
import io.gaia_app.registries.SourceRepository;
import io.gaia_app.registries.github.GithubRegistryApi;
import io.gaia_app.registries.gitlab.GitlabRegistryApi;
import org.springframework.stereotype.Service;

import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import java.util.stream.Stream;

@Service
public class ModuleServiceImpl implements ModuleService {

/**
* matches https://github.com/something(.git), and captures 'something'.
* The first group (.+?) expands as few times as possible.
* The second non-capturing group (?:) matches the end of the line ($) or .git$
*/
private static final Pattern GITHUB_HTTPS_REPOSITORY_URL_REGEX = Pattern.compile("https://github\\.com/(.+?)(?:(?:$)|(?:\\.git$))");

/**
* matches git@github.com:something(.git), and captures 'something'.
* The first group (.+?) expands as few times as possible.
* The second non-capturing group (?:) matches the end of the line or .git
*/
private static final Pattern GITHUB_SSH_REPOSITORY_URL_REGEX = Pattern.compile("git@github\\.com:(.+?)(?:(?:$)|(?:\\.git$))");

/**
* matches https://gitlab.com/something(.git), and captures 'something'.
* The first group (.+?) expands as few times as possible.
* The second non-capturing group (?:) matches the end of the line or .git
*/
private static final Pattern GITLAB_HTTPS_REPOSITORY_URL_REGEX = Pattern.compile("https://gitlab\\.com/(.+?)(?:(?:/?$)|(?:\\.git$))");

/**
* matches git@gitlab.com:something(.git), and captures 'something'.
* The first group (.+?) expands as few times as possible.
* The second non-capturing group (?:) matches the end of the line or .git
*/
private static final Pattern GITLAB_SSH_REPOSITORY_URL_REGEX = Pattern.compile("git@gitlab\\.com:(.+?)(?:(?:$)|(?:\\.git$))");

private Map<RegistryType, RegistryApi<? extends SourceRepository>> registryApis;

public ModuleServiceImpl(GithubRegistryApi githubRegistryApi, GitlabRegistryApi gitlabRegistryApiMap) {
this.registryApis = Map.of(RegistryType.GITHUB, githubRegistryApi, RegistryType.GITLAB, gitlabRegistryApiMap);
}

@Override
public void updateRegistryDetails(TerraformModule module) {
RegistryType registryType;
if (module.getGitRepositoryUrl().contains("github")) {
registryType = RegistryType.GITHUB;
} else if (module.getGitRepositoryUrl().contains("gitlab")) {
registryType = RegistryType.GITLAB;
} else {
// could not match any registry type from the URL
return;
}
// find first pattern that matches, then computes the details
Stream.of(
GITHUB_HTTPS_REPOSITORY_URL_REGEX,
GITHUB_SSH_REPOSITORY_URL_REGEX,
GITLAB_HTTPS_REPOSITORY_URL_REGEX,
GITLAB_SSH_REPOSITORY_URL_REGEX)
.map(pattern -> pattern.matcher(module.getGitRepositoryUrl()))
.filter(Matcher::matches)
.findFirst()
.ifPresent(it -> {
var projectName = it.group(1);
// get project id from project name
var repository = registryApis.get(registryType).getRepository(projectName);
module.setRegistryDetails(new RegistryDetails(registryType, repository.getId()));
});
}
}
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
package io.gaia_app.modules.controller;

import io.gaia_app.modules.ModuleService;
import io.gaia_app.modules.bo.TerraformModule;
import io.gaia_app.modules.repository.TerraformModuleGitRepository;
import io.gaia_app.modules.repository.TerraformModuleRepository;
Expand Down Expand Up @@ -31,11 +32,14 @@ public class ModuleRestController {

private RegistryService registryService;

private ModuleService moduleService;

@Autowired
public ModuleRestController(TerraformModuleRepository moduleRepository, TerraformModuleGitRepository moduleGitRepository, RegistryService registryService) {
public ModuleRestController(TerraformModuleRepository moduleRepository, TerraformModuleGitRepository moduleGitRepository, RegistryService registryService, ModuleService moduleService) {
this.moduleRepository = moduleRepository;
this.moduleGitRepository = moduleGitRepository;
this.registryService = registryService;
this.moduleService = moduleService;
}

@GetMapping
Expand Down Expand Up @@ -73,6 +77,9 @@ public TerraformModule saveModule(@PathVariable String id, @RequestBody @Valid T
throw new ModuleForbiddenException();
}

// try to update registry details
moduleService.updateRegistryDetails(module);

module.getModuleMetadata().setUpdatedBy(user);
module.getModuleMetadata().setUpdatedAt(LocalDateTime.now());

Expand Down
45 changes: 32 additions & 13 deletions src/main/java/io/gaia_app/registries/AbstractRegistryApi.kt
Original file line number Diff line number Diff line change
Expand Up @@ -4,36 +4,44 @@ import io.gaia_app.teams.User
import org.springframework.http.HttpEntity
import org.springframework.http.HttpHeaders
import org.springframework.http.HttpMethod
import org.springframework.http.HttpStatus
import org.springframework.web.client.HttpClientErrorException
import org.springframework.web.client.RestTemplate
import java.net.http.HttpClient
import java.util.*

abstract class AbstractRegistryApi<K: SourceRepository>(val restTemplate: RestTemplate,
private val registryType: RegistryType,
private val registryFileClass: Class<K>,
private val registryListFileClass: Class<Array<K>>): RegistryApi<K> {
abstract class AbstractRegistryApi<K : SourceRepository>(val restTemplate: RestTemplate,
private val registryType: RegistryType,
private val registryFileClass: Class<K>,
private val registryListFileClass: Class<Array<K>>) : RegistryApi<K> {

private fun <T> callWithAuth(url: String, token: String, responseType: Class<T>): T? {
fun <T> callWithoutAuth(url: String, responseType: Class<T>): T? {
try {
val response = restTemplate.getForEntity(
url,
responseType)
return response.body
} catch (e: HttpClientErrorException) {
// in case of 404, returns null as an empty body
return null
}
}

fun <T> callWithAuth(url: String, token: String, responseType: Class<T>): T? {
val headers = HttpHeaders()
headers.add("Authorization", "Bearer $token")

val requestEntity = HttpEntity<Any>(headers)

try{
try {
val response = restTemplate.exchange(
url,
HttpMethod.GET,
requestEntity,
responseType)
return response.body
}
catch (e:HttpClientErrorException){
} catch (e: HttpClientErrorException) {
// in case of 404, returns null as an empty body
return null
}

}

override fun getRepositories(user: User): List<K> {
Expand All @@ -45,6 +53,13 @@ abstract class AbstractRegistryApi<K: SourceRepository>(val restTemplate: RestTe
return callWithAuth(url, token, registryListFileClass)?.toList() ?: emptyList()
}

override fun getRepository(projectId: String): K {
// fetching repositories
val url = registryType.repositoryUrl.format(projectId)

return callWithoutAuth(url, registryFileClass)!!
}

override fun getRepository(user: User, projectId: String): K {
// fetching repositories
val url = registryType.repositoryUrl.format(projectId)
Expand All @@ -57,9 +72,13 @@ abstract class AbstractRegistryApi<K: SourceRepository>(val restTemplate: RestTe
override fun getFileContent(user: User, projectId: String, filename: String): String {
val url = registryType.fileContentUrl.format(projectId, filename)

val token = user.oAuth2User?.token!!
val token = user.oAuth2User?.token

val file = callWithAuth(url, token, RegistryFile::class.java)
val file = if (token != null) {
callWithAuth(url, token, RegistryFile::class.java)
} else {
callWithoutAuth(url, RegistryFile::class.java)
}

// removing trailing \n
return String(Base64.getDecoder().decode(file?.content?.replace("\n", "") ?: ""))
Expand Down
8 changes: 8 additions & 0 deletions src/main/java/io/gaia_app/registries/RegistryApi.kt
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,16 @@ interface RegistryApi<T> {

fun getRepositories(user: User) : List<T>

/**
* Gets a repository data for an authenticated user
*/
fun getRepository(user: User, projectId: String): T

/**
* Gets a repository data for an anonymous user
*/
fun getRepository(projectId: String): T

fun getFileContent(user: User, projectId: String, filename: String): String

}

0 comments on commit 57e49d3

Please sign in to comment.