Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques…#1393
Merged
jdcasey merged 35 commits intoCommonjava:masterfrom Nov 4, 2019
geored:NOS-2048
Merged
Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques…#1393jdcasey merged 35 commits intoCommonjava:masterfrom geored:NOS-2048
jdcasey merged 35 commits intoCommonjava:masterfrom
geored:NOS-2048
Conversation
…ts for browsing content - NOS-2048.Also i have rewrite @Head method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception.
Adding Dropwizard Metrics adapter to publish metrics to Prometheus via servlet, since our perf-test instance can't deploy GraphiteDB to a pod. This gets us ready for Prometheus anyway. We may not need content indexing anymore in our particular deployment environment, and it might actually be hurting performance for our use case. Adding an option to disable it via config.
Allow disable of content index, and add prometheus metrics adapter
jdcasey
reviewed
Oct 21, 2019
Member
jdcasey
left a comment
jdcasey
left a comment
There was a problem hiding this comment.
Do we have functional tests to ensure the browse resource for root directory and a subdirectory both work? I'm a little concerned that we're commenting out those separate methods here, unless we have tests that verify both use cases.
| @Inject | ||
| ContentDigester contentDigester; | ||
|
|
||
| @Context UriInfo uriInfo; |
Member
There was a problem hiding this comment.
Do we need to remove @ApplicationScoped from the class for this to work in a multi-threaded way?
Member
Author
There was a problem hiding this comment.
Right now its not problem multi-threaded that much as returning
"HTTP/1.1 500 Internal Server Error" on every each http HEAD request to that endpoint ("/api/browse/{packageType}/{type}/{name}/{path: (.*)}") , that's way i'm changing to one method which will handle both routes ( "/" and "/{path}/{path}/... ) with this setup.
For tests yes , i will provide them on next commit iteration for this PR
Member
There was a problem hiding this comment.
I'm just wondering if we should inject these @context fields as method parameters to avoid cross-request pollution since the class is application scoped.
Member
There was a problem hiding this comment.
As far as the tests, if you're still working on this can you mark it as WIP? When the tests come up clean it can be confusing whether it's ready to merge or not.
Member
Author
There was a problem hiding this comment.
"I'm just wondering if we should inject these @context fields as method parameters to avoid cross-request pollution since the class is application scoped."
- it is one to one relation with every request, problem is ( what i noticed and i search but so far didn't find any specific reason to back up my claim ) when it is put as parameter reference ( im not sure is it @context or is it @httpservletrequest who is doing this ) inside @Head http request it is creating exception ( relating to "i cannot find relating info" this maybe is only specific for our environment ) and thats way i remove it rom @Head and place it on object ( which basically should be same ) but on other hand maybe you are right when saying that @applicationscoped is causing thread polution acros requests because we have one instance for each request....
Yes , i will remove @ApplicationScoped and try without and let you know.
Member
Author
There was a problem hiding this comment.
"As far as the tests, if you're still working on this can you mark it as WIP? When the tests come up clean it can be confusing whether it's ready to merge or not."
John there is other NOS-1896 issue which is addresing same task but it is broather in sense that is asking for content checkup and deletion based on this MD5/SHA1 header checkup.
Sorry for not cleaned code , i have noticed latter on and i must do that because of hurry.
Member
Author
There was a problem hiding this comment.
Yes @jdcasey @Applicationscope is creating server problems with @context injected inside method as parameter , also second @Head method with @path("/") is generating server error even without @applicationscope. One other issue is that pathmapping ( @path("/{path (.*)} ) is returning :"415 Media Unsuported " response.
…variables for Grafana
Improve Prometheus metrics using node_prefix as a label to help with Grafana variables
Reuse the existing repo as implied one should add it into group
…ype}/{type}/{name}/{path (.*)}'
… Folo tracking records
… so we can use Externalizable versioning for deser
Refine deployment from Jenkinsfile, add timestamps for access in Folo tracked entries
jdcasey
requested changes
Oct 30, 2019
Member
jdcasey
left a comment
jdcasey
left a comment
There was a problem hiding this comment.
okay, so I just noticed that the REST code changes are in ContentBrowseResource, which is just concerned with generating directory listings. All of the links to actual concrete files actually directs the user to the content-access resource classes: MavenContentAccessResource, NpmContentAccessResource, etc.
So, I started up a test Indy instance, then requested a file from Maven Central: http://localhost:8080/api/content/maven/remote/central/org/commonjava/commonjava/12/commonjava-12.pom ...then, I ran a HEAD request for this same file:
$ curl -I http://localhost:8080/api/content/maven/remote/central/org/commonjava/commonjava/12/commonjava-12.pom
HTTP/1.1 200 OK
Indy-Origin: maven:remote:central
ACCEPT-RANGES: bytes
DATE: Wed, 30 Oct 2019 21:59:44 GMT
CONNECTION: keep-alive
X-CHECKSUM-SHA1: 4fd28d272b6c97eb307d09dea44390ed30a5ef5b
AGE: 5368244
X-CACHE-HITS: 1, 1
X-CHECKSUM-MD5: cbe550218f592381f72d594fdf970cc6
Indy-Min-API-Version: 0.0
VIA: 1.1 varnish
ETAG: "cbe550218f592381f72d594fdf970cc6"
LAST-MODIFIED: Mon, 27 Feb 2017 21:56:07 GMT
Indy-Cur-API-Version: 1
X-SERVED-BY: cache-iad2151-IAD, cache-atl6234-ATL
X-TIMER: S1572472785.762667,VS0,VE1
X-CACHE: HIT, HIT
CONTENT-LENGTH: 19234
Content-Type: text/xml
I realized this was using the http-metadata.json file to fill in the headers from Maven Central, so I deleted that file from storage and re-ran the command:
$ curl -I http://localhost:8080/api/content/maven/remote/central/org/commonjava/commonjava/12/commonjava-12.pom
HTTP/1.1 200 OK
Indy-Min-API-Version: 0.0
Connection: keep-alive
Last-Modified: Wed, 30 Oct 2019 16:59:44 GMT
Indy-Cur-API-Version: 1
Indy-Origin: maven:remote:central
Content-Length: 19234
Content-Type: application/maven+xml
Date: Wed, 30 Oct 2019 22:06:43 GMT
I believe this is the REST resource we should be improving with X-CHECKSUM-SHA1 and X-CHECKSUM-MD5 headers, for consistency with Maven Central. It won't give us perfect consistency with all remote repos, but it'll at least align our code with central.
IOW, I think changing ContentBrowseResource isn't really what we need, since it only generates directory listing JSON...when the checksums we want are for concrete files. If we wanted to, we could also improve the ListingURL DTO to include checksum info. Then, we wouldn't need to issue secondary HEAD requests to Indy.
Deprecated listContents in client module as it is not working now
Member
Author
Yes , i have noticed that also , thats way i have place this md5 & sha1 header returning here - because this will be basically done for every file that doesn't have provided sha1/md5 headers ( so we can be sure that every file is with hash information ) and for those that we do have those hashes will be our primary source of comparison. This is basically no harm solution because if it is not content file then this headers will be "null" and as such will not be placed in header map and if it is content file then first we will check for "X-CHECKSUM-[..]" if that is not available then we check "ETAG" and if both of them are not available then as last solution we check provided headers from this PR which will be based on keystore value from repository. |
Member
Author
|
... Upon examination again i have conclude that problem with my reasoning is that i was going from assumption that every request made to /api/content/... is redirected to /api/browse/... but now i see that logic is applied only if path is directory and not file, if it is a file then request is processed by "ContentAccessHandler". |
Member
Author
.... Also other problem is that on first "HEAD" request for content this response is returned:$ http HEAD http://localhost:8080/api/content/maven/remote/central/org/commonjava/commonjava/12/commonjava-12.pom
|
NOS-2099:
jdcasey
approved these changes
Nov 4, 2019
| responseHelper.setInfoHeaders( builder, item, sk, path, true, contentType, | ||
| httpMetadata ); | ||
|
|
||
| if(!path.endsWith("/")) { |
Member
There was a problem hiding this comment.
You had this in ResponseHelper before...what prompted you to move it into ContentAccessHandler? Was it causing problems with some other REST class?
Member
Author
There was a problem hiding this comment.
Yes , some of integration test have made errors [RecalculateMissingChecksumForTrackingRecordTest] on build so that's way i move it it here in ContentAcessHandler.
ruhan1
added a commit
that referenced
this pull request
Nov 8, 2019
* Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Bump swagger-ui from 2.2.10 to 3.23.11 in /uis/layover Bumps [swagger-ui](https://github.com/swagger-api/swagger-ui) from 2.2.10 to 3.23.11. - [Release notes](https://github.com/swagger-api/swagger-ui/releases) - [Commits](swagger-api/swagger-ui@v2.2.10...v3.23.11) Signed-off-by: dependabot[bot] <support@github.com> * turn off deflate encoding support until we understand why it can corrupt downloaded archives * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques… (#1393) * Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD requests for browsing content - NOS-2048.Also i have rewrite @Head method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception. * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Fix Bad Request 400 error when uploading snapshot maven-metadata (#1389) * Allow disable of content index, and add prometheus metrics adapter Adding Dropwizard Metrics adapter to publish metrics to Prometheus via servlet, since our perf-test instance can't deploy GraphiteDB to a pod. This gets us ready for Prometheus anyway. We may not need content indexing anymore in our particular deployment environment, and it might actually be hurting performance for our use case. Adding an option to disable it via config. * add managed dep on indy-subsys-metrics-prometheus * Fix tests broken by default-disabled content index, and improve some logger names * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Added Tests for checking http HEAD requests on '/api/browse/{packageType}/{type}/{name}/{path (.*)}' * Changes Requested * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Path not directory * Content Digest if path doesn't end with '/' * Added headers in ContentAcessHandler
ruhan1
added a commit
that referenced
this pull request
Nov 8, 2019
* Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Bump swagger-ui from 2.2.10 to 3.23.11 in /uis/layover Bumps [swagger-ui](https://github.com/swagger-api/swagger-ui) from 2.2.10 to 3.23.11. - [Release notes](https://github.com/swagger-api/swagger-ui/releases) - [Commits](swagger-api/swagger-ui@v2.2.10...v3.23.11) Signed-off-by: dependabot[bot] <support@github.com> * turn off deflate encoding support until we understand why it can corrupt downloaded archives * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques… (#1393) * Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD requests for browsing content - NOS-2048.Also i have rewrite @Head method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception. * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Fix Bad Request 400 error when uploading snapshot maven-metadata (#1389) * Allow disable of content index, and add prometheus metrics adapter Adding Dropwizard Metrics adapter to publish metrics to Prometheus via servlet, since our perf-test instance can't deploy GraphiteDB to a pod. This gets us ready for Prometheus anyway. We may not need content indexing anymore in our particular deployment environment, and it might actually be hurting performance for our use case. Adding an option to disable it via config. * add managed dep on indy-subsys-metrics-prometheus * Fix tests broken by default-disabled content index, and improve some logger names * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Added Tests for checking http HEAD requests on '/api/browse/{packageType}/{type}/{name}/{path (.*)}' * Changes Requested * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Path not directory * Content Digest if path doesn't end with '/' * Added headers in ContentAcessHandler * Integrate with pathmapped storage (#1396) * Integrate with pathmapped storage * Start embedded cassandra in AbstractIndyFunctionalTest * Add missing configs * Start cassandra before fixture server (#1398) * Fix ftest about how to check storage file existence (#1400) * Fix ftest cassandra keyspace and use random port (#1401) * Fix ftest cassandra keyspace and use random port * Fix another two ftests for how to check physical file * Fix getStorageFile by find the latest under hashed dir * Fix some ftests where the initBaseConfig were overriden * Apply cassandra-maven-plugin for ftests (#1402) * Apply cassandra-maven-plugin for ftests * Fix another four ftests via getPhysicalPath * Fix the remaining ftests which were failed due to the change to file check (#1404) * Add subsys/cassandra for general connection settings (#1411) * Add subsys/cassandra for general connection settings * Fix GroupMetadataFileDeletionTest the way it checks file exists * Remove test keyspace to release resources * Handle connect error * Ignore drop keyspace errors * Create pathDB with Cassandra client session * Drop keyspace before session closed
ruhan1
added a commit
that referenced
this pull request
Dec 11, 2019
* Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Bump swagger-ui from 2.2.10 to 3.23.11 in /uis/layover Bumps [swagger-ui](https://github.com/swagger-api/swagger-ui) from 2.2.10 to 3.23.11. - [Release notes](https://github.com/swagger-api/swagger-ui/releases) - [Commits](swagger-api/swagger-ui@v2.2.10...v3.23.11) Signed-off-by: dependabot[bot] <support@github.com> * turn off deflate encoding support until we understand why it can corrupt downloaded archives * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques… (#1393) * Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD requests for browsing content - NOS-2048.Also i have rewrite @Head method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception. * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Fix Bad Request 400 error when uploading snapshot maven-metadata (#1389) * Allow disable of content index, and add prometheus metrics adapter Adding Dropwizard Metrics adapter to publish metrics to Prometheus via servlet, since our perf-test instance can't deploy GraphiteDB to a pod. This gets us ready for Prometheus anyway. We may not need content indexing anymore in our particular deployment environment, and it might actually be hurting performance for our use case. Adding an option to disable it via config. * add managed dep on indy-subsys-metrics-prometheus * Fix tests broken by default-disabled content index, and improve some logger names * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Added Tests for checking http HEAD requests on '/api/browse/{packageType}/{type}/{name}/{path (.*)}' * Changes Requested * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Path not directory * Content Digest if path doesn't end with '/' * Added headers in ContentAcessHandler * switch to using openshift api to control/follow image build (#1415) * switch to using openshift api to control/follow image build * fix hook url var to my_bc in Jenkinsfile * provide better feedback about bc selection process * typo in stage end brace * Start path map storage migrator (#1416) * Start path map storage migrator * Fix deps and remove first slash of store path * Bump jacksonVersion from 2.9.10 to 2.10.1 Bumps `jacksonVersion` from 2.9.10 to 2.10.1. Updates `jackson-core` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson-core/releases) - [Commits](FasterXML/jackson-core@jackson-core-2.9.10...jackson-core-2.10.1) Updates `jackson-annotations` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) Updates `jackson-databind` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) Updates `jackson-jaxrs-json-provider` from 2.9.10 to 2.10.1 Signed-off-by: dependabot[bot] <support@github.com> * Add dedupe check support for pathmap migrator * Use move instead of delete for processed path files in todo folder * Add reload processed count for next migrate from failure (#1423) * Fix the path mapped migrator checksum issue (#1424) * Fix TrackedContentEntryTest (#1425) * Avoid one not needed traverse of first level package path And concurrently run scan from package level * Fix groovy.lang.MissingPropertyException: No such property: tarball_url (#1428) * Add filter to filter out the files not need to migrate * [migration-tool] shutdown pathDB after done (#1430) * Remove pathmap-migrator (#1432) Pathmap-migrator has been moved to a single repo here: https://www.github.com/Commonjava/pathmap-migrator * Remove left pathmap-migrator class * Fix the swagger-ui issue when upgraded to 3.23.x (#1435) * Bump keycloakVersion from 6.0.0 to 7.0.1 (#1410) Bumps `keycloakVersion` from 6.0.0 to 7.0.1. Updates `keycloak-undertow-adapter` from 6.0.0 to 7.0.1 Updates `keycloak-core` from 6.0.0 to 7.0.1 - [Release notes](https://github.com/keycloak/keycloak/releases) - [Commits](keycloak/keycloak@6.0.0...7.0.1) Updates `keycloak-adapter-core` from 6.0.0 to 7.0.1 - [Release notes](https://github.com/keycloak/keycloak/releases) - [Commits](keycloak/keycloak@6.0.0...7.0.1) Signed-off-by: dependabot[bot] <support@github.com> * NOS-2167: Handling bug issue in StoreValidator where process hang if remote repository is not valid (#1438) * provide better feedback about bc selection process * typo in stage end brace * record and pass on data tarball to image build in addition to skinny tarball * remove merge error; duplicate else clause * Remove two dup lines * adding trace logging to figure out why raw timings aren't being metered (#1445) * Accumulate timing info for different types of calls in a given user request, then report these in the REST end log (#1446) * skip all tests during deployment (they run earlier) (#1447) * Fix cumulative timing measurement and add accumulation for non-intercepted metrics (#1448) * Add creation time for all repos Seems there is no creation time for all of our current repos. I think it is useful for tracking, especially for the auto-created repos like koji-proxy and implied repos. BTW, I'm also thinking if we also need to add the "last_update_time" for all repos. * Level down metricsManager log for accumulating to trace * Make repo create time as string for ser/de-ser consideration * Switch to calculating cumulative elapsed milliseconds instead of seconds (#1451) * add license headers and bump to release of galley * [maven-release-plugin] prepare release indy-parent-1.9.6 * [maven-release-plugin] prepare for next development iteration * Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager (#1455) * Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager Previously, storing ArtifactStore definitions as JSON requires us to serialize/deserialize (render and parse) constantly, every time we access a repository definition. We can access dozens or even thousands of these in a request, especially in an upload request or a promotion request. Looking at our current production deployment, we see that upload latency is up around 7 seconds at times, with about 6.98 seconds of that being consumed in the process of working with the store-definition cache. This change implements Externalizable for all ArtifactStore derivatives, StoreKey, and related classes in the core model of Indy. It adds some testing to ensure these can be read / written appropriately using readExternal / writeExternal methods, and verifies that round-trip data is preserved. Then, it switches the cache used by InfinispanStoreDataManager to use Cache<StoreKey, ArtifactStore> instead of Cache<StoreKey, String>, and relies on Infinispan itself to use the Externalizable implementations to serialize data when it is persisted to the backing database. Along with this comes a need to migrate existing JSON data to the new binary format. To handle this, I've modified the InfinispanStoreDataMigrationAction class to look for a cache named 'store-data-v1' and if it finds it, to migrate the old-style store definitions from JSON to the new cache, managed by InfinispanStoreDataManager. It will NOT use the really old filesystem storage location if it finds this legacy cache. * make no-arg constructors public and correct usage of Object[In/Out]putStream * Add mixed-lot serialization for ArtifactStore, and read all by ArtifactStore base class This will test that polymorphism works in the object stream. * Cumulative call count in logging MDC and metering ratio (#1457) * remove unused serialization code * implement cumulative counts for meters, and add support for metering ratios to cut down on metering latency * Using store-data-v1 for new object-based store-data cache (#1456) I found a problem for migration process. When we use db-based persistence for store cache, we can not control the table name for which store use by applying a simple "table-name" like attr in ispn xml because it does not support this. The table name is applied by its cache name directly. So we need to use a new cache name for the new obj-based store, ane use the legacy "store-data" cache for migration purpose. * change the current store-data cache name to 'store-data-v2' to imply that 'store-data' is v1 (#1458)
ruhan1
added a commit
that referenced
this pull request
Dec 17, 2019
* Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Bump swagger-ui from 2.2.10 to 3.23.11 in /uis/layover Bumps [swagger-ui](https://github.com/swagger-api/swagger-ui) from 2.2.10 to 3.23.11. - [Release notes](https://github.com/swagger-api/swagger-ui/releases) - [Commits](swagger-api/swagger-ui@v2.2.10...v3.23.11) Signed-off-by: dependabot[bot] <support@github.com> * turn off deflate encoding support until we understand why it can corrupt downloaded archives * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques… (#1393) * Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD requests for browsing content - NOS-2048.Also i have rewrite @Head method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception. * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Fix Bad Request 400 error when uploading snapshot maven-metadata (#1389) * Allow disable of content index, and add prometheus metrics adapter Adding Dropwizard Metrics adapter to publish metrics to Prometheus via servlet, since our perf-test instance can't deploy GraphiteDB to a pod. This gets us ready for Prometheus anyway. We may not need content indexing anymore in our particular deployment environment, and it might actually be hurting performance for our use case. Adding an option to disable it via config. * add managed dep on indy-subsys-metrics-prometheus * Fix tests broken by default-disabled content index, and improve some logger names * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Added Tests for checking http HEAD requests on '/api/browse/{packageType}/{type}/{name}/{path (.*)}' * Changes Requested * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Path not directory * Content Digest if path doesn't end with '/' * Added headers in ContentAcessHandler * switch to using openshift api to control/follow image build (#1415) * switch to using openshift api to control/follow image build * fix hook url var to my_bc in Jenkinsfile * provide better feedback about bc selection process * typo in stage end brace * Start path map storage migrator (#1416) * Start path map storage migrator * Fix deps and remove first slash of store path * Bump jacksonVersion from 2.9.10 to 2.10.1 Bumps `jacksonVersion` from 2.9.10 to 2.10.1. Updates `jackson-core` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson-core/releases) - [Commits](FasterXML/jackson-core@jackson-core-2.9.10...jackson-core-2.10.1) Updates `jackson-annotations` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) Updates `jackson-databind` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) Updates `jackson-jaxrs-json-provider` from 2.9.10 to 2.10.1 Signed-off-by: dependabot[bot] <support@github.com> * Add dedupe check support for pathmap migrator * Use move instead of delete for processed path files in todo folder * Add reload processed count for next migrate from failure (#1423) * Fix the path mapped migrator checksum issue (#1424) * Fix TrackedContentEntryTest (#1425) * Avoid one not needed traverse of first level package path And concurrently run scan from package level * Fix groovy.lang.MissingPropertyException: No such property: tarball_url (#1428) * Add filter to filter out the files not need to migrate * [migration-tool] shutdown pathDB after done (#1430) * Remove pathmap-migrator (#1432) Pathmap-migrator has been moved to a single repo here: https://www.github.com/Commonjava/pathmap-migrator * Remove left pathmap-migrator class * Fix the swagger-ui issue when upgraded to 3.23.x (#1435) * Bump keycloakVersion from 6.0.0 to 7.0.1 (#1410) Bumps `keycloakVersion` from 6.0.0 to 7.0.1. Updates `keycloak-undertow-adapter` from 6.0.0 to 7.0.1 Updates `keycloak-core` from 6.0.0 to 7.0.1 - [Release notes](https://github.com/keycloak/keycloak/releases) - [Commits](keycloak/keycloak@6.0.0...7.0.1) Updates `keycloak-adapter-core` from 6.0.0 to 7.0.1 - [Release notes](https://github.com/keycloak/keycloak/releases) - [Commits](keycloak/keycloak@6.0.0...7.0.1) Signed-off-by: dependabot[bot] <support@github.com> * NOS-2167: Handling bug issue in StoreValidator where process hang if remote repository is not valid (#1438) * provide better feedback about bc selection process * typo in stage end brace * record and pass on data tarball to image build in addition to skinny tarball * remove merge error; duplicate else clause * Remove two dup lines * adding trace logging to figure out why raw timings aren't being metered (#1445) * Accumulate timing info for different types of calls in a given user request, then report these in the REST end log (#1446) * skip all tests during deployment (they run earlier) (#1447) * Fix cumulative timing measurement and add accumulation for non-intercepted metrics (#1448) * Add creation time for all repos Seems there is no creation time for all of our current repos. I think it is useful for tracking, especially for the auto-created repos like koji-proxy and implied repos. BTW, I'm also thinking if we also need to add the "last_update_time" for all repos. * Level down metricsManager log for accumulating to trace * Make repo create time as string for ser/de-ser consideration * Switch to calculating cumulative elapsed milliseconds instead of seconds (#1451) * add license headers and bump to release of galley * [maven-release-plugin] prepare release indy-parent-1.9.6 * [maven-release-plugin] prepare for next development iteration * Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager (#1455) * Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager Previously, storing ArtifactStore definitions as JSON requires us to serialize/deserialize (render and parse) constantly, every time we access a repository definition. We can access dozens or even thousands of these in a request, especially in an upload request or a promotion request. Looking at our current production deployment, we see that upload latency is up around 7 seconds at times, with about 6.98 seconds of that being consumed in the process of working with the store-definition cache. This change implements Externalizable for all ArtifactStore derivatives, StoreKey, and related classes in the core model of Indy. It adds some testing to ensure these can be read / written appropriately using readExternal / writeExternal methods, and verifies that round-trip data is preserved. Then, it switches the cache used by InfinispanStoreDataManager to use Cache<StoreKey, ArtifactStore> instead of Cache<StoreKey, String>, and relies on Infinispan itself to use the Externalizable implementations to serialize data when it is persisted to the backing database. Along with this comes a need to migrate existing JSON data to the new binary format. To handle this, I've modified the InfinispanStoreDataMigrationAction class to look for a cache named 'store-data-v1' and if it finds it, to migrate the old-style store definitions from JSON to the new cache, managed by InfinispanStoreDataManager. It will NOT use the really old filesystem storage location if it finds this legacy cache. * make no-arg constructors public and correct usage of Object[In/Out]putStream * Add mixed-lot serialization for ArtifactStore, and read all by ArtifactStore base class This will test that polymorphism works in the object stream. * Cumulative call count in logging MDC and metering ratio (#1457) * remove unused serialization code * implement cumulative counts for meters, and add support for metering ratios to cut down on metering latency * Using store-data-v1 for new object-based store-data cache (#1456) I found a problem for migration process. When we use db-based persistence for store cache, we can not control the table name for which store use by applying a simple "table-name" like attr in ispn xml because it does not support this. The table name is applied by its cache name directly. So we need to use a new cache name for the new obj-based store, ane use the legacy "store-data" cache for migration purpose. * change the current store-data cache name to 'store-data-v2' to imply that 'store-data' is v1 (#1458) * Drill down metrics for store.execute (#1460) * Remove old secrets loading code in indy.sh (#1453) * Rewrite store-query methods that use stream() and intercept/wrap store-query for metrics (#1461) * Rewrite store-query methods that use stream() and intercept/wrap store-query for metrics We cannot use the @measure annotation in ArtifactStoreQuery, since that class is instantiated a variable number of calls for each user request, and instances hold state supporting the fluent API. The StoreDataManager decorator wraps ArtifactStoreQuery results from the query() method in a query class that measures query calls, delegating the actual query to the instance that comes from the underlying StoreDataManager delegate. StoreDataManager stream / getAll methods rely on having in-memory access to ALL of the ArtifactStore instances, and also on having a small enough instance count to make iterating through all of them feasible. In our production environment, both of these assumptions are violated. We need a better way to access this information, but to start I've refactored DefaultArtifactStoreQuery methods that use streams and getAll in the StoreDataManager so they won't do that. In the case of retrieving the stores that are in the membership tree of a particular group, I've rewritten that to avoid recursion. In addition, I've added a keyStream() method to ArtifactStoreQuery and a streamArtifactStoreKeys() method to StoreDataManager. This SHOULD be something that we can optimize, ensuring all keys are available in-memory or easy to deserialize with minimum overhead. We can do a lot of filtering based on the key alone, then additional filtering can be applied with the matching stores, which would be retrieved directly by key from the filtered keys in these calls. * remove duplicate dependencies in 2 poms * Separate metered-db stuff to break dependency cycle, and fix a few compilation / unit test problems * Add license headers * [maven-release-plugin] prepare release indy-parent-1.9.7 * [maven-release-plugin] prepare for next development iteration * Add tck for ispn store data manager And following: * One more tck case for complex group hierarchy for affected by testing * Removed commented code in infinispan data store manager for pretty * FIX: NPE happened during getAffectedBy Seems the new refactored code brings some new way to get affected groups, but it missed some NPE check which caused 500 error. * update license headers in new classes * [maven-release-plugin] prepare release indy-parent-1.9.7.1 * [maven-release-plugin] prepare for next development iteration * Update snapshot version to 1.9.8 * Fix mistakes
ruhan1
added a commit
that referenced
this pull request
Feb 3, 2020
* Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Bump swagger-ui from 2.2.10 to 3.23.11 in /uis/layover Bumps [swagger-ui](https://github.com/swagger-api/swagger-ui) from 2.2.10 to 3.23.11. - [Release notes](https://github.com/swagger-api/swagger-ui/releases) - [Commits](swagger-api/swagger-ui@v2.2.10...v3.23.11) Signed-off-by: dependabot[bot] <support@github.com> * turn off deflate encoding support until we understand why it can corrupt downloaded archives * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques… (#1393) * Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD requests for browsing content - NOS-2048.Also i have rewrite @Head method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception. * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Fix Bad Request 400 error when uploading snapshot maven-metadata (#1389) * Allow disable of content index, and add prometheus metrics adapter Adding Dropwizard Metrics adapter to publish metrics to Prometheus via servlet, since our perf-test instance can't deploy GraphiteDB to a pod. This gets us ready for Prometheus anyway. We may not need content indexing anymore in our particular deployment environment, and it might actually be hurting performance for our use case. Adding an option to disable it via config. * add managed dep on indy-subsys-metrics-prometheus * Fix tests broken by default-disabled content index, and improve some logger names * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Added Tests for checking http HEAD requests on '/api/browse/{packageType}/{type}/{name}/{path (.*)}' * Changes Requested * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Path not directory * Content Digest if path doesn't end with '/' * Added headers in ContentAcessHandler * switch to using openshift api to control/follow image build (#1415) * switch to using openshift api to control/follow image build * fix hook url var to my_bc in Jenkinsfile * provide better feedback about bc selection process * typo in stage end brace * Start path map storage migrator (#1416) * Start path map storage migrator * Fix deps and remove first slash of store path * Bump jacksonVersion from 2.9.10 to 2.10.1 Bumps `jacksonVersion` from 2.9.10 to 2.10.1. Updates `jackson-core` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson-core/releases) - [Commits](FasterXML/jackson-core@jackson-core-2.9.10...jackson-core-2.10.1) Updates `jackson-annotations` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) Updates `jackson-databind` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) Updates `jackson-jaxrs-json-provider` from 2.9.10 to 2.10.1 Signed-off-by: dependabot[bot] <support@github.com> * Add dedupe check support for pathmap migrator * Use move instead of delete for processed path files in todo folder * Add reload processed count for next migrate from failure (#1423) * Fix the path mapped migrator checksum issue (#1424) * Fix TrackedContentEntryTest (#1425) * Avoid one not needed traverse of first level package path And concurrently run scan from package level * Fix groovy.lang.MissingPropertyException: No such property: tarball_url (#1428) * Add filter to filter out the files not need to migrate * [migration-tool] shutdown pathDB after done (#1430) * Remove pathmap-migrator (#1432) Pathmap-migrator has been moved to a single repo here: https://www.github.com/Commonjava/pathmap-migrator * Remove left pathmap-migrator class * Fix the swagger-ui issue when upgraded to 3.23.x (#1435) * Bump keycloakVersion from 6.0.0 to 7.0.1 (#1410) Bumps `keycloakVersion` from 6.0.0 to 7.0.1. Updates `keycloak-undertow-adapter` from 6.0.0 to 7.0.1 Updates `keycloak-core` from 6.0.0 to 7.0.1 - [Release notes](https://github.com/keycloak/keycloak/releases) - [Commits](keycloak/keycloak@6.0.0...7.0.1) Updates `keycloak-adapter-core` from 6.0.0 to 7.0.1 - [Release notes](https://github.com/keycloak/keycloak/releases) - [Commits](keycloak/keycloak@6.0.0...7.0.1) Signed-off-by: dependabot[bot] <support@github.com> * NOS-2167: Handling bug issue in StoreValidator where process hang if remote repository is not valid (#1438) * provide better feedback about bc selection process * typo in stage end brace * record and pass on data tarball to image build in addition to skinny tarball * remove merge error; duplicate else clause * Remove two dup lines * adding trace logging to figure out why raw timings aren't being metered (#1445) * Accumulate timing info for different types of calls in a given user request, then report these in the REST end log (#1446) * skip all tests during deployment (they run earlier) (#1447) * Fix cumulative timing measurement and add accumulation for non-intercepted metrics (#1448) * Add creation time for all repos Seems there is no creation time for all of our current repos. I think it is useful for tracking, especially for the auto-created repos like koji-proxy and implied repos. BTW, I'm also thinking if we also need to add the "last_update_time" for all repos. * Level down metricsManager log for accumulating to trace * Make repo create time as string for ser/de-ser consideration * Switch to calculating cumulative elapsed milliseconds instead of seconds (#1451) * add license headers and bump to release of galley * [maven-release-plugin] prepare release indy-parent-1.9.6 * [maven-release-plugin] prepare for next development iteration * Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager (#1455) * Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager Previously, storing ArtifactStore definitions as JSON requires us to serialize/deserialize (render and parse) constantly, every time we access a repository definition. We can access dozens or even thousands of these in a request, especially in an upload request or a promotion request. Looking at our current production deployment, we see that upload latency is up around 7 seconds at times, with about 6.98 seconds of that being consumed in the process of working with the store-definition cache. This change implements Externalizable for all ArtifactStore derivatives, StoreKey, and related classes in the core model of Indy. It adds some testing to ensure these can be read / written appropriately using readExternal / writeExternal methods, and verifies that round-trip data is preserved. Then, it switches the cache used by InfinispanStoreDataManager to use Cache<StoreKey, ArtifactStore> instead of Cache<StoreKey, String>, and relies on Infinispan itself to use the Externalizable implementations to serialize data when it is persisted to the backing database. Along with this comes a need to migrate existing JSON data to the new binary format. To handle this, I've modified the InfinispanStoreDataMigrationAction class to look for a cache named 'store-data-v1' and if it finds it, to migrate the old-style store definitions from JSON to the new cache, managed by InfinispanStoreDataManager. It will NOT use the really old filesystem storage location if it finds this legacy cache. * make no-arg constructors public and correct usage of Object[In/Out]putStream * Add mixed-lot serialization for ArtifactStore, and read all by ArtifactStore base class This will test that polymorphism works in the object stream. * Cumulative call count in logging MDC and metering ratio (#1457) * remove unused serialization code * implement cumulative counts for meters, and add support for metering ratios to cut down on metering latency * Using store-data-v1 for new object-based store-data cache (#1456) I found a problem for migration process. When we use db-based persistence for store cache, we can not control the table name for which store use by applying a simple "table-name" like attr in ispn xml because it does not support this. The table name is applied by its cache name directly. So we need to use a new cache name for the new obj-based store, ane use the legacy "store-data" cache for migration purpose. * change the current store-data cache name to 'store-data-v2' to imply that 'store-data' is v1 (#1458) * Drill down metrics for store.execute (#1460) * Remove old secrets loading code in indy.sh (#1453) * Rewrite store-query methods that use stream() and intercept/wrap store-query for metrics (#1461) * Rewrite store-query methods that use stream() and intercept/wrap store-query for metrics We cannot use the @measure annotation in ArtifactStoreQuery, since that class is instantiated a variable number of calls for each user request, and instances hold state supporting the fluent API. The StoreDataManager decorator wraps ArtifactStoreQuery results from the query() method in a query class that measures query calls, delegating the actual query to the instance that comes from the underlying StoreDataManager delegate. StoreDataManager stream / getAll methods rely on having in-memory access to ALL of the ArtifactStore instances, and also on having a small enough instance count to make iterating through all of them feasible. In our production environment, both of these assumptions are violated. We need a better way to access this information, but to start I've refactored DefaultArtifactStoreQuery methods that use streams and getAll in the StoreDataManager so they won't do that. In the case of retrieving the stores that are in the membership tree of a particular group, I've rewritten that to avoid recursion. In addition, I've added a keyStream() method to ArtifactStoreQuery and a streamArtifactStoreKeys() method to StoreDataManager. This SHOULD be something that we can optimize, ensuring all keys are available in-memory or easy to deserialize with minimum overhead. We can do a lot of filtering based on the key alone, then additional filtering can be applied with the matching stores, which would be retrieved directly by key from the filtered keys in these calls. * remove duplicate dependencies in 2 poms * Separate metered-db stuff to break dependency cycle, and fix a few compilation / unit test problems * Add license headers * [maven-release-plugin] prepare release indy-parent-1.9.7 * [maven-release-plugin] prepare for next development iteration * Add tck for ispn store data manager And following: * One more tck case for complex group hierarchy for affected by testing * Removed commented code in infinispan data store manager for pretty * FIX: NPE happened during getAffectedBy Seems the new refactored code brings some new way to get affected groups, but it missed some NPE check which caused 500 error. * update license headers in new classes * [maven-release-plugin] prepare release indy-parent-1.9.7.1 * [maven-release-plugin] prepare for next development iteration * Remove unused jdk internal imports * Add comment for new group mangement tck case * Extract getStore operation from deep loop for affected by This getArtifactStore resides in a deep loop, which will cause serious performance problem if the loop size is very huge(Actually it is). So need to extract it out from this loop. * Async the group de-index during upload in content-index service Recent performance problem we met for uploading & promotion is all caused by the getAffectedBy operation, which is a time-consumed one for a full scan on store data. After checking, I found that it is mainly used for de-index or re-index content-index entries for the uploading repo. I think this de-index will not be a block factor during the upload or promotion process, which means we could make it async. In this commit, I created a ISPN cache and used it like a queue, which decouple the de-index opeartion from storing process in content-index to an aync listener of ISPN listener. Why not use a weft executor? Because the storing operation is high frequent, which means it could be blocked again if the de-index effeciency is much slower than the real physical storing operation. Use this queue-liked way we can delegate all these to underlying ISPN and we have a position to set the queue size from ISPN config. * Revert "Async the group de-index during upload in content-index service" This reverts commit 0b3ef58. With load-testing, found that there are a bunch of ispn lock acquiring timeout error, so that means the getAffectedBy operation is much slower than file transferring itself, and cause many unhandled de-index entries in that cache. We need to think about another solution to switch to async mode. * Async the group de-index during upload in content-index service Recent performance problem we met for uploading & promotion is all caused by the getAffectedBy operation, which is a time-consumed one for a full scan on store data. After checking, I found that it is mainly used for de-index or re-index content-index entries for the uploading repo. I think this de-index will not be a block factor during the upload or promotion process, which means we could make it async. In this commit, I switched back to use a normal executor service to async the de-index of the affected group. One potential problem for this solution is the executor queue size, which could cause OOM if there are bunch of uploading happened in a short time period. We need to monitor if this can happen. * Code cleanup: remove dupe and unused deps (#1474) Purpose to reduce indy launcher tarball size. Did following: * Removed all duped junit, harmcrest and cdi delaration * Removed quartz deps as it is not used now * Removed h2 as it is not used now * don't build cache migrator or record extractor in main build. (#1478) These don't need to be on the main release cycle, and it seems there's some recursion problem in the maven-assembly-plugin with one of them that's screwing up Indy builds. * Thread-off two nfc cleaning ops which refers getAffectedBy (#1477) * Thread-off getAffectedBy ops for nfc cleaning in ContentManager * Thread-off getAffectedBy ops for nfc cleaning in promote validation * register NFC cache listener in DefaultContentIndexManager (#1479) * Limit "affected-by" with full scan opeartion in package and type based set (#1471) Add a new Cache which is <package, Map<type, Set<StoreKey>> to store all Store keys based on package and type, so for affected group get method, we can avoid the full scan on all repos to get groups, but just simplified to two map get ops. * Fix NPE when store is null in case of being deleted * Thread-off getAffectedBy ops in NFCContentListener * Extract getGroupsAffectedBy from Query to StoreDataManager * Add reverse map for affected stores by some constituents change * Limit "affected-by" with full scan opeartion in package and type based set (#1471) Add a new Cache which is <package, Map<type, Set<StoreKey>> to store all Store keys based on package and type, so for affected group get method, we can avoid the full scan on all repos to get groups, but just simplified to two map get ops. * Simplify affected-by cache We won't completely flatten out the affected-by cache by resolving groups affected by groups affected by a given store. Instead, we leave it as a simple reverse map of the group's constituents, which will give us direct keys to lookup in order to iterate and calculated these n-level affected relationships. It's still much better than scanning all groups (which in the past meant scanning all repos). At the same time, it will avoid any unforseen problems with group membership changes that don't get propagated via the flattening process. It should be easier to maintain. * populate the affected-by cache on startup if the affected-by cache is clear This should work if the affected-by cache is kept in memory, or for the first run when we use a database to store it. We're lining this up with the by-package-type cache as a single-node fix, where we will still need a more scalable solution from a proper database / data design approach before we're ready for multi-node usage. * clean up affected-by iteration and consolidate iteration in default store query I added a null check and disabled-group check to the iteration for finding groups affected by a particular store. The rest of these are just adjusting away from streams and toward intermediary Set instances...things like that. * fix merge error in StoreDataManager * fix remaining compilation errors left from merging * fix NPE when ssl validation repo is null due to some error * Fix potential NPE by using PackageMetadataMerger as a component, as intended. * Update copyright year range to include 2020, reformat license headers (#1484) * fix license headers * Update project copyright year range to include 2020 and reformat license headers * [maven-release-plugin] prepare release indy-parent-1.9.8 * [maven-release-plugin] prepare for next development iteration * Fix some code smell for PromotionManager like potential NPE Conflicts: api/src/main/java/org/commonjava/indy/data/StoreDataManager.java * Some code simplify Conflicts: db/common/src/main/java/org/commonjava/indy/db/common/AbstractStoreDataManager.java * Code optimization - MavenMetadataGenerator * Simplify lambda * Remove useless null check * Use try resource instead * Charset standardization * Add some trace log for metadata clearing of pom update (#1487) * Fix init of affected-by reverse map, and add checks/error message if it's empty (#1488) * Fix affected-by reversemap initialization in lifecycle startup * Add logging to note when the affected-by cache is altered, and another error log message if it's empty * Refactor InfinispanStoreDataManager.init() to startup actions, add some logging The init() method runs before any migration actions run, which means migrated stores won't be stored in the byPkg cache. The old initializer for the affected-by map has a similar problem, so I removed it in favor of the new startup action. The byPkg init was moved out to a startup action too for this reason. Finally, I've added a log message to show which groups were found to be affected by a specific set of input keys. This will give us better information about whether the affected-by map is being used or not. * adding another ftest that initializes store data at boot instead of at migration * Support honeycomb * Fix config * Add HoneycombInterceptor * Fix getchildspan * Add HoneycombFilter * Add pathInfo to fields * Fix NPE in Honeycomb filter and add some fields to honeycomb spans * Enable collection of more metrics (esp. I/O level) via Honeycomb (#1492) This required refactoring the Galley timing function providers, using a new patch of Galley that attempts to generalize the metrics interface, at least for I/O operations. It also involved creating three new interceptors, which would handle methods that are designed to wrap arbitrary pieces of code in metrics. * return the Honeycomb trace span at the filter level, not the root span that is inside it. (#1493) * add golden signals filter into IndyDeployment because filter ordering is important (#1494) * fix cyclic dependency caused by moving GoldenSignalsFilter (#1495) * fix cyclic dependency caused by moving GoldenSignalsFilter * Adding beans.xml to allow CDI injection properly * upgrade galley to release and fix license headers * [maven-release-plugin] prepare release indy-parent-1.9.9 * [maven-release-plugin] prepare for next development iteration * Honeycomb field-capture improvements, and span configuration (#1497) * Fix the Honeycomb root span (which I broke previously) * debug some honeycomb field-tracking problems I had to break out the ThreadContext initialization into an outer servlet filter (specified first in the mapping order, since it's the outer one) to preserve fields until honeycomb can find it. I've also added the ability to turn spans on/off in the configuration, using EITHER include OR exclude semantics (you can't use both). * ignore metrics testing for now...test needs to be updated * Consolidate all async getGroupAffectedBy ops in StoreDataManager * New Relic telemetry support (#1498) * Add New Relic telemetry support, and complete configuration of interceptors in beans.xml for Honeycomb * check before sending batch, push field extraction to manager, use Dequeue instead of List * Add New Relic telemetry support, and complete configuration of interceptors in beans.xml for Honeycomb * check before sending batch, push field extraction to manager, use Dequeue instead of List * Refine / fix new relic span tracking * try to forget we ever used New Relic tracing, the UI is terrible * Add new configuration approach for honeycomb and configurable sampling rates Sampling rates can be specified by Golden Signals (SLI) function name, or by method name, or by class name + method name, etc. * forgot to add configuration parameter for base sample rate * fix compilation error * Tie in threaded-off calls to Honeycomb trace by simulating jump to another service * refactored thread contextualizer for honeycomb I'm ignoring the client bound to other threads and just creating a new one from config when needed. This should work around NPEs in deeper / uncontrolled threads. * Align version to 1.9.10-SNAPSHOT * Update license head * Fix merge issues * Fix merge issues-2 * Fix 2 ftests where the bytestream is drained after the first use * Revert "Fix 2 ftests where the bytestream is drained after the first use" This reverts commit 81df08a. Co-authored-by: Wenjie Guo <guowj12@gmail.com> Co-authored-by: georgy <3187346+gorgija@users.noreply.github.com> Co-authored-by: Gang Li <ligangty@gmail.com> Co-authored-by: John Casey <jdcasey@commonjava.org> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: geored <47973913+geored@users.noreply.github.com>
ruhan1
added a commit
that referenced
this pull request
Feb 12, 2020
* Integrate with pathmapped storage (#1396)
* Integrate with pathmapped storage
* Start embedded cassandra in AbstractIndyFunctionalTest
* Add missing configs
* Start cassandra before fixture server (#1398)
* Fix ftest about how to check storage file existence (#1400)
* Fix ftest cassandra keyspace and use random port (#1401)
* Fix ftest cassandra keyspace and use random port
* Fix another two ftests for how to check physical file
* Fix getStorageFile by find the latest under hashed dir
* Fix some ftests where the initBaseConfig were overriden
* Apply cassandra-maven-plugin for ftests (#1402)
* Apply cassandra-maven-plugin for ftests
* Fix another four ftests via getPhysicalPath
* Fix the remaining ftests which were failed due to the change to file check (#1404)
* Add subsys/cassandra for general connection settings (#1411)
* Add subsys/cassandra for general connection settings
* Fix GroupMetadataFileDeletionTest the way it checks file exists
* Remove test keyspace to release resources
* Handle connect error
* Ignore drop keyspace errors
* Create pathDB with Cassandra client session
* Drop keyspace before session closed
* Master integration pathmap mergemaster (#1412)
* Reuse the existing repo as implied one should add it into group
* NOS-2099:
Added information from validating RemoteRepositories inside RemoteRepository metadata object.
Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service.
* Test
* Filter the existing repos matching the conditions
* Enhance the condition for comparing the repo
* Deprecated listContents in client module as it is not working now
* Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana
* Tweak the condition and merge the keys of metadata "implied_by_stores"
* Changes Required
* only do deploy when we're doing an image build, and add timestamps to Folo tracking records
* Adding unit tests for version change in TrackedContentEntry
* Bug fix
* InternalFeature isn't null value
* DefaultStoreValidatorTest
* Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser
* Bump swagger-ui from 2.2.10 to 3.23.11 in /uis/layover
Bumps [swagger-ui](https://github.com/swagger-api/swagger-ui) from 2.2.10 to 3.23.11.
- [Release notes](https://github.com/swagger-api/swagger-ui/releases)
- [Commits](https://github.com/swagger-api/swagger-ui/compare/v2.2.10...v3.23.11)
Signed-off-by: dependabot[bot] <support@github.com>
* turn off deflate encoding support until we understand why it can corrupt downloaded archives
* Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques… (#1393)
* Reuse the existing repo as implied one should add it into group
* NOS-2099:
Added information from validating RemoteRepositories inside RemoteRepository metadata object.
Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service.
* Test
* Added MD5 and SHA1 digested hashes inside header for HTTP HEAD requests for browsing content - NOS-2048.Also i have rewrite @HEAD method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception.
* Filter the existing repos matching the conditions
* Enhance the condition for comparing the repo
* Fix Bad Request 400 error when uploading snapshot maven-metadata (#1389)
* Allow disable of content index, and add prometheus metrics adapter
Adding Dropwizard Metrics adapter to publish metrics to Prometheus via servlet, since our perf-test instance can't deploy GraphiteDB to a pod. This gets us ready for Prometheus anyway.
We may not need content indexing anymore in our particular deployment environment, and it might actually be hurting performance for our use case. Adding an option to disable it via config.
* add managed dep on indy-subsys-metrics-prometheus
* Fix tests broken by default-disabled content index, and improve some logger names
* Deprecated listContents in client module as it is not working now
* Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana
* Tweak the condition and merge the keys of metadata "implied_by_stores"
* Added Tests for checking http HEAD requests on '/api/browse/{packageType}/{type}/{name}/{path (.*)}'
* Changes Requested
* Changes Required
* only do deploy when we're doing an image build, and add timestamps to Folo tracking records
* Adding unit tests for version change in TrackedContentEntry
* Bug fix
* InternalFeature isn't null value
* DefaultStoreValidatorTest
* Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser
* Added Digesting Algos for Repository contents inside 'ResponseHelper' class
* Added Digesting Algos for Repository contents inside 'ResponseHelper' class
* Path not directory
* Content Digest if path doesn't end with '/'
* Added headers in ContentAcessHandler
* Master integration pathmap rebase (#1413)
* Reuse the existing repo as implied one should add it into group
* NOS-2099:
Added information from validating RemoteRepositories inside RemoteRepository metadata object.
Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service.
* Test
* Filter the existing repos matching the conditions
* Enhance the condition for comparing the repo
* Deprecated listContents in client module as it is not working now
* Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana
* Tweak the condition and merge the keys of metadata "implied_by_stores"
* Changes Required
* only do deploy when we're doing an image build, and add timestamps to Folo tracking records
* Adding unit tests for version change in TrackedContentEntry
* Bug fix
* InternalFeature isn't null value
* DefaultStoreValidatorTest
* Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser
* Bump swagger-ui from 2.2.10 to 3.23.11 in /uis/layover
Bumps [swagger-ui](https://github.com/swagger-api/swagger-ui) from 2.2.10 to 3.23.11.
- [Release notes](https://github.com/swagger-api/swagger-ui/releases)
- [Commits](https://github.com/swagger-api/swagger-ui/compare/v2.2.10...v3.23.11)
Signed-off-by: dependabot[bot] <support@github.com>
* turn off deflate encoding support until we understand why it can corrupt downloaded archives
* Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques… (#1393)
* Reuse the existing repo as implied one should add it into group
* NOS-2099:
Added information from validating RemoteRepositories inside RemoteRepository metadata object.
Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service.
* Test
* Added MD5 and SHA1 digested hashes inside header for HTTP HEAD requests for browsing content - NOS-2048.Also i have rewrite @HEAD method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception.
* Filter the existing repos matching the conditions
* Enhance the condition for comparing the repo
* Fix Bad Request 400 error when uploading snapshot maven-metadata (#1389)
* Allow disable of content index, and add prometheus metrics adapter
Adding Dropwizard Metrics adapter to publish metrics to Prometheus via servlet, since our perf-test instance can't deploy GraphiteDB to a pod. This gets us ready for Prometheus anyway.
We may not need content indexing anymore in our particular deployment environment, and it might actually be hurting performance for our use case. Adding an option to disable it via config.
* add managed dep on indy-subsys-metrics-prometheus
* Fix tests broken by default-disabled content index, and improve some logger names
* Deprecated listContents in client module as it is not working now
* Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana
* Tweak the condition and merge the keys of metadata "implied_by_stores"
* Added Tests for checking http HEAD requests on '/api/browse/{packageType}/{type}/{name}/{path (.*)}'
* Changes Requested
* Changes Required
* only do deploy when we're doing an image build, and add timestamps to Folo tracking records
* Adding unit tests for version change in TrackedContentEntry
* Bug fix
* InternalFeature isn't null value
* DefaultStoreValidatorTest
* Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser
* Added Digesting Algos for Repository contents inside 'ResponseHelper' class
* Added Digesting Algos for Repository contents inside 'ResponseHelper' class
* Path not directory
* Content Digest if path doesn't end with '/'
* Added headers in ContentAcessHandler
* Integrate with pathmapped storage (#1396)
* Integrate with pathmapped storage
* Start embedded cassandra in AbstractIndyFunctionalTest
* Add missing configs
* Start cassandra before fixture server (#1398)
* Fix ftest about how to check storage file existence (#1400)
* Fix ftest cassandra keyspace and use random port (#1401)
* Fix ftest cassandra keyspace and use random port
* Fix another two ftests for how to check physical file
* Fix getStorageFile by find the latest under hashed dir
* Fix some ftests where the initBaseConfig were overriden
* Apply cassandra-maven-plugin for ftests (#1402)
* Apply cassandra-maven-plugin for ftests
* Fix another four ftests via getPhysicalPath
* Fix the remaining ftests which were failed due to the change to file check (#1404)
* Add subsys/cassandra for general connection settings (#1411)
* Add subsys/cassandra for general connection settings
* Fix GroupMetadataFileDeletionTest the way it checks file exists
* Remove test keyspace to release resources
* Handle connect error
* Ignore drop keyspace errors
* Create pathDB with Cassandra client session
* Drop keyspace before session closed
* Measure pathDB operations (#1414)
* Adapt to path-mapped package changes (#1417)
* Add confset for subsys/cassandra (#1419)
* Adapt to path-mapped package changes
* Add confset for subsys/cassandra
* switch to using openshift api to control/follow image build (#1415) (#1418)
* switch to using openshift api to control/follow image build
* fix hook url var to my_bc in Jenkinsfile
* provide better feedback about bc selection process
* typo in stage end brace
* Set cassandraClient so-connection timeout and add close method (#1422)
* Add CassandraClient.close
* Add a debug log
* Extend cassandra cluster maxMem
* Back to standalone cassandra
* Put off close
* Try again by only close session
* use cassandraClient.close
* Fix TrackedContentEntryTest
* setConnectTimeoutMillis to 30 seconds
* Fix groovy.lang.MissingPropertyException: No such property: tarball_url (#1427)
* Fix npe in CassandraClient (#1431)
* Fix npe in CassandraClient
* Set socket options
* Addon path-mapped to provide direct listing/get/delete operations (#1436)
* Cherry fixes for Swagger and bump some versions (#1437)
* Fix the swagger-ui issue when upgraded to 3.23.x (#1435)
* Bump keycloakVersion from 6.0.0 to 7.0.1 (#1410)
Bumps `keycloakVersion` from 6.0.0 to 7.0.1.
Updates `keycloak-undertow-adapter` from 6.0.0 to 7.0.1
Updates `keycloak-core` from 6.0.0 to 7.0.1
- [Release notes](https://github.com/keycloak/keycloak/releases)
- [Commits](https://github.com/keycloak/keycloak/compare/6.0.0...7.0.1)
Updates `keycloak-adapter-core` from 6.0.0 to 7.0.1
- [Release notes](https://github.com/keycloak/keycloak/releases)
- [Commits](https://github.com/keycloak/keycloak/compare/6.0.0...7.0.1)
Signed-off-by: dependabot[bot] <support@github.com>
* Add query param limit to pathmapresource.list (#1439)
* Filter listing result by query param type (file,dir,all) (#1441)
* Filter listing result by query param type (file,dir,all)
* Update the filter
* Remove useless import
* Update
* Fix null fileType
* Merge master to path-mapped feature branch (#1459)
* Reuse the existing repo as implied one should add it into group
* NOS-2099:
Added information from validating RemoteRepositories inside RemoteRepository metadata object.
Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service.
* Test
* Filter the existing repos matching the conditions
* Enhance the condition for comparing the repo
* Deprecated listContents in client module as it is not working now
* Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana
* Tweak the condition and merge the keys of metadata "implied_by_stores"
* Changes Required
* only do deploy when we're doing an image build, and add timestamps to Folo tracking records
* Adding unit tests for version change in TrackedContentEntry
* Bug fix
* InternalFeature isn't null value
* DefaultStoreValidatorTest
* Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser
* Bump swagger-ui from 2.2.10 to 3.23.11 in /uis/layover
Bumps [swagger-ui](https://github.com/swagger-api/swagger-ui) from 2.2.10 to 3.23.11.
- [Release notes](https://github.com/swagger-api/swagger-ui/releases)
- [Commits](https://github.com/swagger-api/swagger-ui/compare/v2.2.10...v3.23.11)
Signed-off-by: dependabot[bot] <support@github.com>
* turn off deflate encoding support until we understand why it can corrupt downloaded archives
* Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques… (#1393)
* Reuse the existing repo as implied one should add it into group
* NOS-2099:
Added information from validating RemoteRepositories inside RemoteRepository metadata object.
Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service.
* Test
* Added MD5 and SHA1 digested hashes inside header for HTTP HEAD requests for browsing content - NOS-2048.Also i have rewrite @HEAD method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception.
* Filter the existing repos matching the conditions
* Enhance the condition for comparing the repo
* Fix Bad Request 400 error when uploading snapshot maven-metadata (#1389)
* Allow disable of content index, and add prometheus metrics adapter
Adding Dropwizard Metrics adapter to publish metrics to Prometheus via servlet, since our perf-test instance can't deploy GraphiteDB to a pod. This gets us ready for Prometheus anyway.
We may not need content indexing anymore in our particular deployment environment, and it might actually be hurting performance for our use case. Adding an option to disable it via config.
* add managed dep on indy-subsys-metrics-prometheus
* Fix tests broken by default-disabled content index, and improve some logger names
* Deprecated listContents in client module as it is not working now
* Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana
* Tweak the condition and merge the keys of metadata "implied_by_stores"
* Added Tests for checking http HEAD requests on '/api/browse/{packageType}/{type}/{name}/{path (.*)}'
* Changes Requested
* Changes Required
* only do deploy when we're doing an image build, and add timestamps to Folo tracking records
* Adding unit tests for version change in TrackedContentEntry
* Bug fix
* InternalFeature isn't null value
* DefaultStoreValidatorTest
* Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser
* Added Digesting Algos for Repository contents inside 'ResponseHelper' class
* Added Digesting Algos for Repository contents inside 'ResponseHelper' class
* Path not directory
* Content Digest if path doesn't end with '/'
* Added headers in ContentAcessHandler
* switch to using openshift api to control/follow image build (#1415)
* switch to using openshift api to control/follow image build
* fix hook url var to my_bc in Jenkinsfile
* provide better feedback about bc selection process
* typo in stage end brace
* Start path map storage migrator (#1416)
* Start path map storage migrator
* Fix deps and remove first slash of store path
* Bump jacksonVersion from 2.9.10 to 2.10.1
Bumps `jacksonVersion` from 2.9.10 to 2.10.1.
Updates `jackson-core` from 2.9.10 to 2.10.1
- [Release notes](https://github.com/FasterXML/jackson-core/releases)
- [Commits](https://github.com/FasterXML/jackson-core/compare/jackson-core-2.9.10...jackson-core-2.10.1)
Updates `jackson-annotations` from 2.9.10 to 2.10.1
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)
Updates `jackson-databind` from 2.9.10 to 2.10.1
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)
Updates `jackson-jaxrs-json-provider` from 2.9.10 to 2.10.1
Signed-off-by: dependabot[bot] <support@github.com>
* Add dedupe check support for pathmap migrator
* Use move instead of delete for processed path files in todo folder
* Add reload processed count for next migrate from failure (#1423)
* Fix the path mapped migrator checksum issue (#1424)
* Fix TrackedContentEntryTest (#1425)
* Avoid one not needed traverse of first level package path
And concurrently run scan from package level
* Fix groovy.lang.MissingPropertyException: No such property: tarball_url (#1428)
* Add filter to filter out the files not need to migrate
* [migration-tool] shutdown pathDB after done (#1430)
* Remove pathmap-migrator (#1432)
Pathmap-migrator has been moved to a single repo here:
https://www.github.com/Commonjava/pathmap-migrator
* Remove left pathmap-migrator class
* Fix the swagger-ui issue when upgraded to 3.23.x (#1435)
* Bump keycloakVersion from 6.0.0 to 7.0.1 (#1410)
Bumps `keycloakVersion` from 6.0.0 to 7.0.1.
Updates `keycloak-undertow-adapter` from 6.0.0 to 7.0.1
Updates `keycloak-core` from 6.0.0 to 7.0.1
- [Release notes](https://github.com/keycloak/keycloak/releases)
- [Commits](https://github.com/keycloak/keycloak/compare/6.0.0...7.0.1)
Updates `keycloak-adapter-core` from 6.0.0 to 7.0.1
- [Release notes](https://github.com/keycloak/keycloak/releases)
- [Commits](https://github.com/keycloak/keycloak/compare/6.0.0...7.0.1)
Signed-off-by: dependabot[bot] <support@github.com>
* NOS-2167: Handling bug issue in StoreValidator where process hang if remote repository is not valid (#1438)
* provide better feedback about bc selection process
* typo in stage end brace
* record and pass on data tarball to image build in addition to skinny tarball
* remove merge error; duplicate else clause
* Remove two dup lines
* adding trace logging to figure out why raw timings aren't being metered (#1445)
* Accumulate timing info for different types of calls in a given user request, then report these in the REST end log (#1446)
* skip all tests during deployment (they run earlier) (#1447)
* Fix cumulative timing measurement and add accumulation for non-intercepted metrics (#1448)
* Add creation time for all repos
Seems there is no creation time for all of our current repos. I think
it is useful for tracking, especially for the auto-created repos like
koji-proxy and implied repos.
BTW, I'm also thinking if we also need to add the "last_update_time"
for all repos.
* Level down metricsManager log for accumulating to trace
* Make repo create time as string for ser/de-ser consideration
* Switch to calculating cumulative elapsed milliseconds instead of seconds (#1451)
* add license headers and bump to release of galley
* [maven-release-plugin] prepare release indy-parent-1.9.6
* [maven-release-plugin] prepare for next development iteration
* Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager (#1455)
* Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager
Previously, storing ArtifactStore definitions as JSON requires us to serialize/deserialize
(render and parse) constantly, every time we access a repository definition. We can access
dozens or even thousands of these in a request, especially in an upload request or a
promotion request.
Looking at our current production deployment, we see that upload latency is up around 7 seconds
at times, with about 6.98 seconds of that being consumed in the process of working with the
store-definition cache.
This change implements Externalizable for all ArtifactStore derivatives, StoreKey, and related
classes in the core model of Indy. It adds some testing to ensure these can be read / written
appropriately using readExternal / writeExternal methods, and verifies that round-trip data is
preserved.
Then, it switches the cache used by InfinispanStoreDataManager to use Cache<StoreKey, ArtifactStore>
instead of Cache<StoreKey, String>, and relies on Infinispan itself to use the Externalizable
implementations to serialize data when it is persisted to the backing database.
Along with this comes a need to migrate existing JSON data to the new binary format. To handle
this, I've modified the InfinispanStoreDataMigrationAction class to look for a cache named
'store-data-v1' and if it finds it, to migrate the old-style store definitions from JSON to
the new cache, managed by InfinispanStoreDataManager. It will NOT use the really old filesystem
storage location if it finds this legacy cache.
* make no-arg constructors public and correct usage of Object[In/Out]putStream
* Add mixed-lot serialization for ArtifactStore, and read all by ArtifactStore base class
This will test that polymorphism works in the object stream.
* Cumulative call count in logging MDC and metering ratio (#1457)
* remove unused serialization code
* implement cumulative counts for meters, and add support for metering ratios to cut down on metering latency
* Using store-data-v1 for new object-based store-data cache (#1456)
I found a problem for migration process. When we use db-based
persistence for store cache, we can not control the table name for which
store use by applying a simple "table-name" like attr in ispn xml
because it does not support this. The table name is applied by its cache
name directly. So we need to use a new cache name for the new obj-based
store, ane use the legacy "store-data" cache for migration purpose.
* change the current store-data cache name to 'store-data-v2' to imply that 'store-data' is v1 (#1458)
* Merge master to pathmap (1.9.8-SNAPSHOT) (#1464)
* Reuse the existing repo as implied one should add it into group
* NOS-2099:
Added information from validating RemoteRepositories inside RemoteRepository metadata object.
Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service.
* Test
* Filter the existing repos matching the conditions
* Enhance the condition for comparing the repo
* Deprecated listContents in client module as it is not working now
* Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana
* Tweak the condition and merge the keys of metadata "implied_by_stores"
* Changes Required
* only do deploy when we're doing an image build, and add timestamps to Folo tracking records
* Adding unit tests for version change in TrackedContentEntry
* Bug fix
* InternalFeature isn't null value
* DefaultStoreValidatorTest
* Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser
* Bump swagger-ui from 2.2.10 to 3.23.11 in /uis/layover
Bumps [swagger-ui](https://github.com/swagger-api/swagger-ui) from 2.2.10 to 3.23.11.
- [Release notes](https://github.com/swagger-api/swagger-ui/releases)
- [Commits](https://github.com/swagger-api/swagger-ui/compare/v2.2.10...v3.23.11)
Signed-off-by: dependabot[bot] <support@github.com>
* turn off deflate encoding support until we understand why it can corrupt downloaded archives
* Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques… (#1393)
* Reuse the existing repo as implied one should add it into group
* NOS-2099:
Added information from validating RemoteRepositories inside RemoteRepository metadata object.
Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service.
* Test
* Added MD5 and SHA1 digested hashes inside header for HTTP HEAD requests for browsing content - NOS-2048.Also i have rewrite @HEAD method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception.
* Filter the existing repos matching the conditions
* Enhance the condition for comparing the repo
* Fix Bad Request 400 error when uploading snapshot maven-metadata (#1389)
* Allow disable of content index, and add prometheus metrics adapter
Adding Dropwizard Metrics adapter to publish metrics to Prometheus via servlet, since our perf-test instance can't deploy GraphiteDB to a pod. This gets us ready for Prometheus anyway.
We may not need content indexing anymore in our particular deployment environment, and it might actually be hurting performance for our use case. Adding an option to disable it via config.
* add managed dep on indy-subsys-metrics-prometheus
* Fix tests broken by default-disabled content index, and improve some logger names
* Deprecated listContents in client module as it is not working now
* Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana
* Tweak the condition and merge the keys of metadata "implied_by_stores"
* Added Tests for checking http HEAD requests on '/api/browse/{packageType}/{type}/{name}/{path (.*)}'
* Changes Requested
* Changes Required
* only do deploy when we're doing an image build, and add timestamps to Folo tracking records
* Adding unit tests for version change in TrackedContentEntry
* Bug fix
* InternalFeature isn't null value
* DefaultStoreValidatorTest
* Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser
* Added Digesting Algos for Repository contents inside 'ResponseHelper' class
* Added Digesting Algos for Repository contents inside 'ResponseHelper' class
* Path not directory
* Content Digest if path doesn't end with '/'
* Added headers in ContentAcessHandler
* switch to using openshift api to control/follow image build (#1415)
* switch to using openshift api to control/follow image build
* fix hook url var to my_bc in Jenkinsfile
* provide better feedback about bc selection process
* typo in stage end brace
* Start path map storage migrator (#1416)
* Start path map storage migrator
* Fix deps and remove first slash of store path
* Bump jacksonVersion from 2.9.10 to 2.10.1
Bumps `jacksonVersion` from 2.9.10 to 2.10.1.
Updates `jackson-core` from 2.9.10 to 2.10.1
- [Release notes](https://github.com/FasterXML/jackson-core/releases)
- [Commits](https://github.com/FasterXML/jackson-core/compare/jackson-core-2.9.10...jackson-core-2.10.1)
Updates `jackson-annotations` from 2.9.10 to 2.10.1
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)
Updates `jackson-databind` from 2.9.10 to 2.10.1
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)
Updates `jackson-jaxrs-json-provider` from 2.9.10 to 2.10.1
Signed-off-by: dependabot[bot] <support@github.com>
* Add dedupe check support for pathmap migrator
* Use move instead of delete for processed path files in todo folder
* Add reload processed count for next migrate from failure (#1423)
* Fix the path mapped migrator checksum issue (#1424)
* Fix TrackedContentEntryTest (#1425)
* Avoid one not needed traverse of first level package path
And concurrently run scan from package level
* Fix groovy.lang.MissingPropertyException: No such property: tarball_url (#1428)
* Add filter to filter out the files not need to migrate
* [migration-tool] shutdown pathDB after done (#1430)
* Remove pathmap-migrator (#1432)
Pathmap-migrator has been moved to a single repo here:
https://www.github.com/Commonjava/pathmap-migrator
* Remove left pathmap-migrator class
* Fix the swagger-ui issue when upgraded to 3.23.x (#1435)
* Bump keycloakVersion from 6.0.0 to 7.0.1 (#1410)
Bumps `keycloakVersion` from 6.0.0 to 7.0.1.
Updates `keycloak-undertow-adapter` from 6.0.0 to 7.0.1
Updates `keycloak-core` from 6.0.0 to 7.0.1
- [Release notes](https://github.com/keycloak/keycloak/releases)
- [Commits](https://github.com/keycloak/keycloak/compare/6.0.0...7.0.1)
Updates `keycloak-adapter-core` from 6.0.0 to 7.0.1
- [Release notes](https://github.com/keycloak/keycloak/releases)
- [Commits](https://github.com/keycloak/keycloak/compare/6.0.0...7.0.1)
Signed-off-by: dependabot[bot] <support@github.com>
* NOS-2167: Handling bug issue in StoreValidator where process hang if remote repository is not valid (#1438)
* provide better feedback about bc selection process
* typo in stage end brace
* record and pass on data tarball to image build in addition to skinny tarball
* remove merge error; duplicate else clause
* Remove two dup lines
* adding trace logging to figure out why raw timings aren't being metered (#1445)
* Accumulate timing info for different types of calls in a given user request, then report these in the REST end log (#1446)
* skip all tests during deployment (they run earlier) (#1447)
* Fix cumulative timing measurement and add accumulation for non-intercepted metrics (#1448)
* Add creation time for all repos
Seems there is no creation time for all of our current repos. I think
it is useful for tracking, especially for the auto-created repos like
koji-proxy and implied repos.
BTW, I'm also thinking if we also need to add the "last_update_time"
for all repos.
* Level down metricsManager log for accumulating to trace
* Make repo create time as string for ser/de-ser consideration
* Switch to calculating cumulative elapsed milliseconds instead of seconds (#1451)
* add license headers and bump to release of galley
* [maven-release-plugin] prepare release indy-parent-1.9.6
* [maven-release-plugin] prepare for next development iteration
* Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager (#1455)
* Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager
Previously, storing ArtifactStore definitions as JSON requires us to serialize/deserialize
(render and parse) constantly, every time we access a repository definition. We can access
dozens or even thousands of these in a request, especially in an upload request or a
promotion request.
Looking at our current production deployment, we see that upload latency is up around 7 seconds
at times, with about 6.98 seconds of that being consumed in the process of working with the
store-definition cache.
This change implements Externalizable for all ArtifactStore derivatives, StoreKey, and related
classes in the core model of Indy. It adds some testing to ensure these can be read / written
appropriately using readExternal / writeExternal methods, and verifies that round-trip data is
preserved.
Then, it switches the cache used by InfinispanStoreDataManager to use Cache<StoreKey, ArtifactStore>
instead of Cache<StoreKey, String>, and relies on Infinispan itself to use the Externalizable
implementations to serialize data when it is persisted to the backing database.
Along with this comes a need to migrate existing JSON data to the new binary format. To handle
this, I've modified the InfinispanStoreDataMigrationAction class to look for a cache named
'store-data-v1' and if it finds it, to migrate the old-style store definitions from JSON to
the new cache, managed by InfinispanStoreDataManager. It will NOT use the really old filesystem
storage location if it finds this legacy cache.
* make no-arg constructors public and correct usage of Object[In/Out]putStream
* Add mixed-lot serialization for ArtifactStore, and read all by ArtifactStore base class
This will test that polymorphism works in the object stream.
* Cumulative call count in logging MDC and metering ratio (#1457)
* remove unused serialization code
* implement cumulative counts for meters, and add support for metering ratios to cut down on metering latency
* Using store-data-v1 for new object-based store-data cache (#1456)
I found a problem for migration process. When we use db-based
persistence for store cache, we can not control the table name for which
store use by applying a simple "table-name" like attr in ispn xml
because it does not support this. The table name is applied by its cache
name directly. So we need to use a new cache name for the new obj-based
store, ane use the legacy "store-data" cache for migration purpose.
* change the current store-data cache name to 'store-data-v2' to imply that 'store-data' is v1 (#1458)
* Drill down metrics for store.execute (#1460)
* Remove old secrets loading code in indy.sh (#1453)
* Rewrite store-query methods that use stream() and intercept/wrap store-query for metrics (#1461)
* Rewrite store-query methods that use stream() and intercept/wrap store-query for metrics
We cannot use the @Measure annotation in ArtifactStoreQuery, since that class is instantiated a variable number of
calls for each user request, and instances hold state supporting the fluent API. The StoreDataManager decorator
wraps ArtifactStoreQuery results from the query() method in a query class that measures query calls, delegating the
actual query to the instance that comes from the underlying StoreDataManager delegate.
StoreDataManager stream / getAll methods rely on having in-memory access to ALL of the ArtifactStore instances,
and also on having a small enough instance count to make iterating through all of them feasible. In our production
environment, both of these assumptions are violated. We need a better way to access this information, but to start
I've refactored DefaultArtifactStoreQuery methods that use streams and getAll in the StoreDataManager so they won't
do that.
In the case of retrieving the stores that are in the membership tree of a particular group, I've rewritten that to
avoid recursion.
In addition, I've added a keyStream() method to ArtifactStoreQuery and a streamArtifactStoreKeys() method to
StoreDataManager. This SHOULD be something that we can optimize, ensuring all keys are available in-memory or easy
to deserialize with minimum overhead. We can do a lot of filtering based on the key alone, then additional
filtering can be applied with the matching stores, which would be retrieved directly by key from the filtered
keys in these calls.
* remove duplicate dependencies in 2 poms
* Separate metered-db stuff to break dependency cycle, and fix a few compilation / unit test problems
* Add license headers
* [maven-release-plugin] prepare release indy-parent-1.9.7
* [maven-release-plugin] prepare for next development iteration
* Add tck for ispn store data manager
And following:
* One more tck case for complex group hierarchy for affected by
testing
* Removed commented code in infinispan data store manager for pretty
* FIX: NPE happened during getAffectedBy
Seems the new refactored code brings some new way to get affected
groups, but it missed some NPE check which caused 500 error.
* update license headers in new classes
* [maven-release-plugin] prepare release indy-parent-1.9.7.1
* [maven-release-plugin] prepare for next development iteration
* Update snapshot version to 1.9.8
* Fix mistakes
* Add path-mapped-storage dep version to indy pom (#1466)
* Make cassandra-maven-plugin run only in ci mode
* Merge master to integration pathmap (#1467)
* Merge master to pathmap branch (#1501)
* Reuse the existing repo as implied one should add it into group
* NOS-2099:
Added information from validating RemoteRepositories inside RemoteRepository metadata object.
Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service.
* Test
* Filter the existing repos matching the conditions
* Enhance the condition for comparing the repo
* Deprecated listContents in client module as it is not working now
* Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana
* Tweak the condition and merge the keys of metadata "implied_by_stores"
* Changes Required
* only do deploy when we're doing an image build, and add timestamps to Folo tracking records
* Adding unit tests for version change in TrackedContentEntry
* Bug fix
* InternalFeature isn't null value
* DefaultStoreValidatorTest
* Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser
* Bump swagger-ui from 2.2.10 to 3.23.11 in /uis/layover
Bumps [swagger-ui](https://github.com/swagger-api/swagger-ui) from 2.2.10 to 3.23.11.
- [Release notes](https://github.com/swagger-api/swagger-ui/releases)
- [Commits](https://github.com/swagger-api/swagger-ui/compare/v2.2.10...v3.23.11)
Signed-off-by: dependabot[bot] <support@github.com>
* turn off deflate encoding support until we understand why it can corrupt downloaded archives
* Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques… (#1393)
* Reuse the existing repo as implied one should add it into group
* NOS-2099:
Added information from validating RemoteRepositories inside RemoteRepository metadata object.
Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service.
* Test
* Added MD5 and SHA1 digested hashes inside header for HTTP HEAD requests for browsing content - NOS-2048.Also i have rewrite @HEAD method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception.
* Filter the existing repos matching the conditions
* Enhance the condition for comparing the repo
* Fix Bad Request 400 error when uploading snapshot maven-metadata (#1389)
* Allow disable of content index, and add prometheus metrics adapter
Adding Dropwizard Metrics adapter to publish metrics to Prometheus via servlet, since our perf-test instance can't deploy GraphiteDB to a pod. This gets us ready for Prometheus anyway.
We may not need content indexing anymore in our particular deployment environment, and it might actually be hurting performance for our use case. Adding an option to disable it via config.
* add managed dep on indy-subsys-metrics-prometheus
* Fix tests broken by default-disabled content index, and improve some logger names
* Deprecated listContents in client module as it is not working now
* Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana
* Tweak the condition and merge the keys of metadata "implied_by_stores"
* Added Tests for checking http HEAD requests on '/api/browse/{packageType}/{type}/{name}/{path (.*)}'
* Changes Requested
* Changes Required
* only do deploy when we're doing an image build, and add timestamps to Folo tracking records
* Adding unit tests for version change in TrackedContentEntry
* Bug fix
* InternalFeature isn't null value
* DefaultStoreValidatorTest
* Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser
* Added Digesting Algos for Repository contents inside 'ResponseHelper' class
* Added Digesting Algos for Repository contents inside 'ResponseHelper' class
* Path not directory
* Content Digest if path doesn't end with '/'
* Added headers in ContentAcessHandler
* switch to using openshift api to control/follow image build (#1415)
* switch to using openshift api to control/follow image build
* fix hook url var to my_bc in Jenkinsfile
* provide better feedback about bc selection process
* typo in stage end brace
* Start path map storage migrator (#1416)
* Start path map storage migrator
* Fix deps and remove first slash of store path
* Bump jacksonVersion from 2.9.10 to 2.10.1
Bumps `jacksonVersion` from 2.9.10 to 2.10.1.
Updates `jackson-core` from 2.9.10 to 2.10.1
- [Release notes](https://github.com/FasterXML/jackson-core/releases)
- [Commits](https://github.com/FasterXML/jackson-core/compare/jackson-core-2.9.10...jackson-core-2.10.1)
Updates `jackson-annotations` from 2.9.10 to 2.10.1
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)
Updates `jackson-databind` from 2.9.10 to 2.10.1
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)
Updates `jackson-jaxrs-json-provider` from 2.9.10 to 2.10.1
Signed-off-by: dependabot[bot] <support@github.com>
* Add dedupe check support for pathmap migrator
* Use move instead of delete for processed path files in todo folder
* Add reload processed count for next migrate from failure (#1423)
* Fix the path mapped migrator checksum issue (#1424)
* Fix TrackedContentEntryTest (#1425)
* Avoid one not needed traverse of first level package path
And concurrently run scan from package level
* Fix groovy.lang.MissingPropertyException: No such property: tarball_url (#1428)
* Add filter to filter out the files not need to migrate
* [migration-tool] shutdown pathDB after done (#1430)
* Remove pathmap-migrator (#1432)
Pathmap-migrator has been moved to a single repo here:
https://www.github.com/Commonjava/pathmap-migrator
* Remove left pathmap-migrator class
* Fix the swagger-ui issue when upgraded to 3.23.x (#1435)
* Bump keycloakVersion from 6.0.0 to 7.0.1 (#1410)
Bumps `keycloakVersion` from 6.0.0 to 7.0.1.
Updates `keycloak-undertow-adapter` from 6.0.0 to 7.0.1
Updates `keycloak-core` from 6.0.0 to 7.0.1
- [Release notes](https://github.com/keycloak/keycloak/releases)
- [Commits](https://github.com/keycloak/keycloak/compare/6.0.0...7.0.1)
Updates `keycloak-adapter-core` from 6.0.0 to 7.0.1
- [Release notes](https://github.com/keycloak/keycloak/releases)
- [Commits](https://github.com/keycloak/keycloak/compare/6.0.0...7.0.1)
Signed-off-by: dependabot[bot] <support@github.com>
* NOS-2167: Handling bug issue in StoreValidator where process hang if remote repository is not valid (#1438)
* provide better feedback about bc selection process
* typo in stage end brace
* record and pass on data tarball to image build in addition to skinny tarball
* remove merge error; duplicate else clause
* Remove two dup lines
* adding trace logging to figure out why raw timings aren't being metered (#1445)
* Accumulate timing info for different types of calls in a given user request, then report these in the REST end log (#1446)
* skip all tests during deployment (they run earlier) (#1447)
* Fix cumulative timing measurement and add accumulation for non-intercepted metrics (#1448)
* Add creation time for all repos
Seems there is no creation time for all of our current repos. I think
it is useful for tracking, especially for the auto-created repos like
koji-proxy and implied repos.
BTW, I'm also thinking if we also need to add the "last_update_time"
for all repos.
* Level down metricsManager log for accumulating to trace
* Make repo create time as string for ser/de-ser consideration
* Switch to calculating cumulative elapsed milliseconds instead of seconds (#1451)
* add license headers and bump to release of galley
* [maven-release-plugin] prepare release indy-parent-1.9.6
* [maven-release-plugin] prepare for next development iteration
* Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager (#1455)
* Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager
Previously, storing ArtifactStore definitions as JSON requires us to serialize/deserialize
(render and parse) constantly, every time we access a repository definition. We can access
dozens or even thousands of these in a request, especially in an upload request or a
promotion request.
Looking at our current production deployment, we see that upload latency is up around 7 seconds
at times, with about 6.98 seconds of that being consumed in the process of working with the
store-definition cache.
This change implements Externalizable for all ArtifactStore derivatives, StoreKey, and related
classes in the core model of Indy. It adds some testing to ensure these can be read / written
appropriately using readExternal / writeExternal methods, and verifies that round-trip data is
preserved.
Then, it switches the cache used by InfinispanStoreDataManager to use Cache<StoreKey, ArtifactStore>
instead of Cache<StoreKey, String>, and relies on Infinispan itself to use the Externalizable
implementations to serialize data when it is persisted to the backing database.
Along with this comes a need to migrate existing JSON data to the new binary format. To handle
this, I've modified the InfinispanStoreDataMigrationAction class to look for a cache named
'store-data-v1' and if it finds it, to migrate the old-style store definitions from JSON to
the new cache, managed by InfinispanStoreDataManager. It will NOT use the really old filesystem
storage location if it finds this legacy cache.
* make no-arg constructors public and correct usage of Object[In/Out]putStream
* Add mixed-lot serialization for ArtifactStore, and read all by ArtifactStore base class
This will test that polymorphism works in the object stream.
* Cumulative call count in logging MDC and metering ratio (#1457)
* remove unused serialization code
* implement cumulative counts for meters, and add support for metering ratios to cut down on metering latency
* Using store-data-v1 for new object-based store-data cache (#1456)
I found a problem for migration process. When we use db-based
persistence for store cache, we can not control the table name for which
store use by applying a simple "table-name" like attr in ispn xml
because it does not support this. The table name is applied by its cache
name directly. So we need to use a new cache name for the new obj-based
store, ane use the legacy "store-data" cache for migration purpose.
* change the current store-data cache name to 'store-data-v2' to imply that 'store-data' is v1 (#1458)
* Drill down metrics for store.execute (#1460)
* Remove old secrets loading code in indy.sh (#1453)
* Rewrite store-query methods that use stream() and intercept/wrap store-query for metrics (#1461)
* Rewrite store-query methods that use stream() and intercept/wrap store-query for metrics
We cannot use the @Measure annotation in ArtifactStoreQuery, since that class is instantiated a variable number of
calls for each user request, and instances hold state supporting the fluent API. The StoreDataManager decorator
wraps ArtifactStoreQuery results from the query() method in a query class that measures query calls, delegating the
actual query to the instance that comes from the underlying StoreDataManager delegate.
StoreDataManager stream / getAll methods rely on having in-memory access to ALL of the ArtifactStore instances,
and also on having a small enough instance count to make iterating through all of them feasible. In our production
environment, both of these assumptions are violated. We need a better way to access this information, but to start
I've refactored DefaultArtifactStoreQuery methods that use streams and getAll in the StoreDataManager so they won't
do that.
In the case of retrieving the stores that are in the membership tree of a particular group, I've rewritten that to
avoid recursion.
In addition, I've added a keyStream() method to ArtifactStoreQuery and a streamArtifactStoreKeys() method to
StoreDataManager. This SHOULD be something that we can optimize, ensuring all keys are available in-memory or easy
to deserialize with minimum overhead. We can do a lot of filtering based on the key alone, then additional
filtering can be applied with the matching stores, which would be retrieved directly by key from the filtered
keys in these calls.
* remove duplicate dependencies in 2 poms
* Separate metered-db stuff to break dependency cycle, and fix a few compilation / unit test problems
* Add license headers
* [maven-release-plugin] prepare release indy-parent-1.9.7
* [maven-release-plugin] prepare for next development iteration
* Add tck for ispn store data manager
And following:
* One more tck case for complex group hierarchy for affected by
testing
* Removed commented code in infinispan data store manager for pretty
* FIX: NPE happened during getAffectedBy
Seems the new refactored code brings some new way to get affected
groups, but it missed some NPE check which caused 500 error.
* update license headers in new classes
* [maven-release-plugin] prepare release indy-parent-1.9.7.1
* [maven-release-plugin] prepare for next development iteration
* Remove unused jdk internal imports
* Add comment for new group mangement tck case
* Extract getStore operation from deep loop for affected by
This getArtifactStore resides in a deep loop, which will cause serious
performance problem if the loop size is very huge(Actually it is). So
need to extract it out from this loop.
* Async the group de-index during upload in content-index service
Recent performance problem we met for uploading & promotion is all
caused by the getAffectedBy operation, which is a time-consumed one for
a full scan on store data. After checking, I found that it is mainly
used for de-index or re-index content-index entries for the uploading
repo. I think this de-index will not be a block factor during the upload
or promotion process, which means we could make it async.
In this commit, I created a ISPN cache and used it like a queue, which
decouple the de-index opeartion from storing process in content-index to
an aync listener of ISPN listener.
Why not use a weft executor? Because the storing operation is high
frequent, which means it could be blocked again if the de-index
effeciency is much slower than the real physical storing operation. Use
this queue-liked way we can delegate all these to underlying ISPN and we
have a position to set the queue size from ISPN config.
* Revert "Async the group de-index during upload in content-index service"
This reverts commit 0b3ef58f054dab2f336d172ba0e839429b339b5e.
With load-testing, found that there are a bunch of ispn lock acquiring
timeout error, so that means the getAffectedBy operation is much
slower than file transferring itself, and cause many unhandled
de-index entries in that cache. We need to think about another
solution to switch to async mode.
* Async the group de-index during upload in content-index service
Recent performance problem we met for uploading & promotion is all
caused by the getAffectedBy operation, which is a time-consumed one for
a full scan on store data. After checking, I found that it is mainly
used for de-index or re-index content-index entries for the uploading
repo. I think this de-index will not be a block factor during the upload
or promotion process, which means we could make it async.
In this commit, I switched back to use a normal executor service to
async the de-index of the affected group. One potential problem for this
solution is the executor queue size, which could cause OOM if there are
bunch of uploading happened in a short time period. We need to monitor
if this can happen.
* Code cleanup: remove dupe and unused deps (#1474)
Purpose to reduce indy launcher tarball size. Did following:
* Removed all duped junit, harmcrest and cdi delaration
* Removed quartz deps as it is not used now
* Removed h2 as it is not used now
* don't build cache migrator or record extractor in main build. (#1478)
These don't need to be on the main release cycle, and it seems there's some recursion
problem in the maven-assembly-plugin with one of them that's screwing up Indy builds.
* Thread-off two nfc cleaning ops which refers getAffectedBy (#1477)
* Thread-off getAffectedBy ops for nfc cleaning in ContentManager
* Thread-off getAffectedBy ops for nfc cleaning in promote validation
* register NFC cache listener in DefaultContentIndexManager (#1479)
* Limit "affected-by" with full scan opeartion in package and type based set (#1471)
Add a new Cache which is <package, Map<type, Set<StoreKey>> to store
all Store keys based on package and type, so for affected group get
method, we can avoid the full scan on all repos to get groups, but just
simplified to two map get ops.
* Fix NPE when store is null in case of being deleted
* Thread-off getAffectedBy ops in NFCContentListener
* Extract getGroupsAffectedBy from Query to StoreDataManager
* Add reverse map for affected stores by some constituents change
* Limit "affected-by" with full scan opeartion in package and type based set (#1471)
Add a new Cache which is <package, Map<type, Set<StoreKey>> to store
all Store keys based on package and type, so for affected group get
method, we can avoid the full scan on all repos to get groups, but just
simplified to two map get ops.
* Simplify affected-by cache
We won't completely flatten out the affected-by cache by resolving groups affected by groups affected by a given store. Instead, we leave it as a simple reverse map of the group's constituents, which will give us direct keys to lookup in order to iterate and calculated these n-level affected relationships. It's still much better than scanning all groups (which in the past meant scanning all repos). At the same time, it will avoid any unforseen problems with group membership changes that don't get propagated via the flattening process. It should be easier to maintain.
* populate the affected-by cache on startup if the affected-by cache is clear
This should work if the affected-by cache is kept in memory, or for the first run when we use a database to
store it. We're lining this up with the by-package-type cache as a single-node fix, where we will still need
a more scalable solution from a proper database / data design approach before we're ready for multi-node usage.
* clean up affected-by iteration and consolidate iteration in default store query
I added a null check and disabled-group check to the iteration for finding groups affected by a particular store.
The rest of these are just adjusting away from streams and toward intermediary Set instances...things like that.
* fix merge error in StoreDataManager
* fix remaining compilation errors left from merging
* fix NPE when ssl validation repo is null due to some error
* Fix potential NPE by using PackageMetadataMerger as a component, as intended.
* Update copyright year range to include 2020, reformat license headers (#1484)
* fix license headers
* Update project copyright year range to include 2020 and reformat license headers
* [maven-release-plugin] prepare release indy-parent-1.9.8
* [maven-release-plugin] prepare for next development iteration
* Fix some code smell for PromotionManager like potential NPE
Conflicts:
api/src/main/java/org/commonjava/indy/data/StoreDataManager.java
* Some code simplify
Conflicts:
db/common/src/main/java/org/commonjava/indy/db/common/AbstractStoreDataManager.java
* Code optimization - MavenMetadataGenerator
* Simplify lambda
* Remove useless null check
* Use try resource instead
* Charset standardization
* Add some trace log for metadata clearing of pom update (#1487)
* Fix init of affected-by reverse map, and add checks/error message if it's empty (#1488)
* Fix affected-by reversemap initialization in lifecycle startup
* Add logging to note when the affected-by cache is altered, and another error log message if it's empty
* Refactor InfinispanStoreDataManager.init() to startup actions, add some logging
The init() method runs before any migration actions run, which means migrated stores won't be stored in the byPkg cache. The old initializer for the affected-by map has a similar problem, so I removed it in favor of the new startup action. The byPkg init was moved out to a startup action too for this reason.
Finally, I've added a log message to show which groups were found to be affected by a specific set of input keys. This will give us better information about whether the affected-by map is being used or not.
* adding another ftest that initializes store data at boot instead of at migration
* Support honeycomb
* Fix config
* Add HoneycombInterceptor
* Fix getchildspan
* Add HoneycombFilter
* Add pathInfo to fields
* Fix NPE in Honeycomb filter and add some fields to honeycomb spans
* Enable collection of more metrics (esp. I/O level) via Honeycomb (#1492)
This required refactoring the Galley timing function providers, using a new patch of Galley that
attempts to generalize the metrics interface, at least for I/O operations.
It also involved creating three new interceptors, which would handle methods that are designed to
wrap arbitrary pieces of code in metrics.
* return the Honeycomb trace span at the filter level, not the root span that is inside it. (#1493)
* add golden signals filter into IndyDeployment because filter ordering is important (#1494)
* fix cyclic dependency caused by moving GoldenSignalsFilter (#1495)
* fix cyclic dependency caused by moving GoldenSignalsFilter
* Adding beans.xml to allow CDI injection properly
* upgrade galley to release and fix license headers
* [maven-release-plugin] prepare release indy-parent-1.9.9
* [maven-release-plugin] prepare for next development iteration
* Honeycomb field-capture improvements, and span configuration (#1497)
* Fix the Honeycomb root span (which I broke previously)
* debug some honeycomb field-tracking problems
I had to break out the ThreadContext initialization into an outer servlet filter (specified first in the mapping order, since it's the outer one) to preserve fields until honeycomb can find it.
I've also added the ability to turn spans on/off in the configuration, using EITHER include OR exclude semantics (you can't use both).
* ignore metrics testing for now...test needs to be updated
* Consolidate all async getGroupAffectedBy ops in StoreDataManager
* New Relic telemetry support (#1498)
* Add New Relic telemetry support, and complete configuration of interceptors in beans.xml for Honeycomb
* check before sending batch, push field extraction to manager, use Dequeue instead of List
* Add New Relic telemetry support, and complete configuration of interceptors in beans.xml for Honeycomb
* check before sending batch, push field extraction to manager, use Dequeue instead of List
* Refine / fix new relic span tracking
* try to forget we ever used New Relic tracing, the UI is terrible
* Add new configuration approach for honeycomb and configurable sampling rates
Sampling rates can be specified by Golden Signals (SLI) function name, or by method name, or by class name + method name, etc.
* forgot to add configuration parameter for base sample rate
* fix compilation error
* Tie in threaded-off calls to Honeycomb trace by simulating jump to another service
* refactored thread contextualizer for honeycomb
I'm ignoring the client bound to other threads and just creating a new one from config when needed. This should work around NPEs in deeper / uncontrolled threads.
* Align version to 1.9.10-SNAPSHOT
* Update license head
* Fix merge issues
* Fix merge issues-2
* Fix 2 ftests where the bytestream is drained after the first use
* Revert "Fix 2 ftests where the bytestream is drained after the first use"
This reverts commit 81df08a2ab3a38d066b65be8b888bbf739c4c223.
Co-authored-by: Wenjie Guo <guowj12@gmail.com>
Co-authored-by: georgy <3187346+gorgija@users.noreply.github.com>
Co-authored-by: Gang Li <ligangty@gmail.com>
Co-authored-by: John Casey <jdcasey@commonjava.org>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: geored <47973913+geored@users.noreply.github.com>
* Fix merge issue
* Solve conflict
* Solve conflict
* Not delete read-only legacy files
* Use constants
Co-authored-by: Gang Li <ligangty@gmail.com>
Co-authored-by: Wenjie Guo <guowj12@gmail.com>
Co-authored-by: georgy <3187346+gorgija@users.noreply.github.com>
Co-authored-by: John Casey <jdcasey@commonjava.org>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: geored <47973913+geored@users.noreply.github.com>
whitingjr
pushed a commit
to whitingjr/indy
that referenced
this pull request
Feb 21, 2020
* Integrate with pathmapped storage (#1396)
* Integrate with pathmapped storage
* Start embedded cassandra in AbstractIndyFunctionalTest
* Add missing configs
* Start cassandra before fixture server (#1398)
* Fix ftest about how to check storage file existence (#1400)
* Fix ftest cassandra keyspace and use random port (#1401)
* Fix ftest cassandra keyspace and use random port
* Fix another two ftests for how to check physical file
* Fix getStorageFile by find the latest under hashed dir
* Fix some ftests where the initBaseConfig were overriden
* Apply cassandra-maven-plugin for ftests (#1402)
* Apply cassandra-maven-plugin for ftests
* Fix another four ftests via getPhysicalPath
* Fix the remaining ftests which were failed due to the change to file check (#1404)
* Add subsys/cassandra for general connection settings (#1411)
* Add subsys/cassandra for general connection settings
* Fix GroupMetadataFileDeletionTest the way it checks file exists
* Remove test keyspace to release resources
* Handle connect error
* Ignore drop keyspace errors
* Create pathDB with Cassandra client session
* Drop keyspace before session closed
* Master integration pathmap mergemaster (#1412)
* Reuse the existing repo as implied one should add it into group
* NOS-2099:
Added information from validating RemoteRepositories inside RemoteRepository metadata object.
Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service.
* Test
* Filter the existing repos matching the conditions
* Enhance the condition for comparing the repo
* Deprecated listContents in client module as it is not working now
* Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana
* Tweak the condition and merge the keys of metadata "implied_by_stores"
* Changes Required
* only do deploy when we're doing an image build, and add timestamps to Folo tracking records
* Adding unit tests for version change in TrackedContentEntry
* Bug fix
* InternalFeature isn't null value
* DefaultStoreValidatorTest
* Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser
* Bump swagger-ui from 2.2.10 to 3.23.11 in /uis/layover
Bumps [swagger-ui](https://github.com/swagger-api/swagger-ui) from 2.2.10 to 3.23.11.
- [Release notes](https://github.com/swagger-api/swagger-ui/releases)
- [Commits](https://github.com/swagger-api/swagger-ui/compare/v2.2.10...v3.23.11)
Signed-off-by: dependabot[bot] <support@github.com>
* turn off deflate encoding support until we understand why it can corrupt downloaded archives
* Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques… (#1393)
* Reuse the existing repo as implied one should add it into group
* NOS-2099:
Added information from validating RemoteRepositories inside RemoteRepository metadata object.
Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service.
* Test
* Added MD5 and SHA1 digested hashes inside header for HTTP HEAD requests for browsing content - NOS-2048.Also i have rewrite @HEAD method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception.
* Filter the existing repos matching the conditions
* Enhance the condition for comparing the repo
* Fix Bad Request 400 error when uploading snapshot maven-metadata (#1389)
* Allow disable of content index, and add prometheus metrics adapter
Adding Dropwizard Metrics adapter to publish metrics to Prometheus via servlet, since our perf-test instance can't deploy GraphiteDB to a pod. This gets us ready for Prometheus anyway.
We may not need content indexing anymore in our particular deployment environment, and it might actually be hurting performance for our use case. Adding an option to disable it via config.
* add managed dep on indy-subsys-metrics-prometheus
* Fix tests broken by default-disabled content index, and improve some logger names
* Deprecated listContents in client module as it is not working now
* Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana
* Tweak the condition and merge the keys of metadata "implied_by_stores"
* Added Tests for checking http HEAD requests on '/api/browse/{packageType}/{type}/{name}/{path (.*)}'
* Changes Requested
* Changes Required
* only do deploy when we're doing an image build, and add timestamps to Folo tracking records
* Adding unit tests for version change in TrackedContentEntry
* Bug fix
* InternalFeature isn't null value
* DefaultStoreValidatorTest
* Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser
* Added Digesting Algos for Repository contents inside 'ResponseHelper' class
* Added Digesting Algos for Repository contents inside 'ResponseHelper' class
* Path not directory
* Content Digest if path doesn't end with '/'
* Added headers in ContentAcessHandler
* Master integration pathmap rebase (#1413)
* Reuse the existing repo as implied one should add it into group
* NOS-2099:
Added information from validating RemoteRepositories inside RemoteRepository metadata object.
Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service.
* Test
* Filter the existing repos matching the conditions
* Enhance the condition for comparing the repo
* Deprecated listContents in client module as it is not working now
* Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana
* Tweak the condition and merge the keys of metadata "implied_by_stores"
* Changes Required
* only do deploy when we're doing an image build, and add timestamps to Folo tracking records
* Adding unit tests for version change in TrackedContentEntry
* Bug fix
* InternalFeature isn't null value
* DefaultStoreValidatorTest
* Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser
* Bump swagger-ui from 2.2.10 to 3.23.11 in /uis/layover
Bumps [swagger-ui](https://github.com/swagger-api/swagger-ui) from 2.2.10 to 3.23.11.
- [Release notes](https://github.com/swagger-api/swagger-ui/releases)
- [Commits](https://github.com/swagger-api/swagger-ui/compare/v2.2.10...v3.23.11)
Signed-off-by: dependabot[bot] <support@github.com>
* turn off deflate encoding support until we understand why it can corrupt downloaded archives
* Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques… (#1393)
* Reuse the existing repo as implied one should add it into group
* NOS-2099:
Added information from validating RemoteRepositories inside RemoteRepository metadata object.
Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service.
* Test
* Added MD5 and SHA1 digested hashes inside header for HTTP HEAD requests for browsing content - NOS-2048.Also i have rewrite @HEAD method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception.
* Filter the existing repos matching the conditions
* Enhance the condition for comparing the repo
* Fix Bad Request 400 error when uploading snapshot maven-metadata (#1389)
* Allow disable of content index, and add prometheus metrics adapter
Adding Dropwizard Metrics adapter to publish metrics to Prometheus via servlet, since our perf-test instance can't deploy GraphiteDB to a pod. This gets us ready for Prometheus anyway.
We may not need content indexing anymore in our particular deployment environment, and it might actually be hurting performance for our use case. Adding an option to disable it via config.
* add managed dep on indy-subsys-metrics-prometheus
* Fix tests broken by default-disabled content index, and improve some logger names
* Deprecated listContents in client module as it is not working now
* Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana
* Tweak the condition and merge the keys of metadata "implied_by_stores"
* Added Tests for checking http HEAD requests on '/api/browse/{packageType}/{type}/{name}/{path (.*)}'
* Changes Requested
* Changes Required
* only do deploy when we're doing an image build, and add timestamps to Folo tracking records
* Adding unit tests for version change in TrackedContentEntry
* Bug fix
* InternalFeature isn't null value
* DefaultStoreValidatorTest
* Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser
* Added Digesting Algos for Repository contents inside 'ResponseHelper' class
* Added Digesting Algos for Repository contents inside 'ResponseHelper' class
* Path not directory
* Content Digest if path doesn't end with '/'
* Added headers in ContentAcessHandler
* Integrate with pathmapped storage (#1396)
* Integrate with pathmapped storage
* Start embedded cassandra in AbstractIndyFunctionalTest
* Add missing configs
* Start cassandra before fixture server (#1398)
* Fix ftest about how to check storage file existence (#1400)
* Fix ftest cassandra keyspace and use random port (#1401)
* Fix ftest cassandra keyspace and use random port
* Fix another two ftests for how to check physical file
* Fix getStorageFile by find the latest under hashed dir
* Fix some ftests where the initBaseConfig were overriden
* Apply cassandra-maven-plugin for ftests (#1402)
* Apply cassandra-maven-plugin for ftests
* Fix another four ftests via getPhysicalPath
* Fix the remaining ftests which were failed due to the change to file check (#1404)
* Add subsys/cassandra for general connection settings (#1411)
* Add subsys/cassandra for general connection settings
* Fix GroupMetadataFileDeletionTest the way it checks file exists
* Remove test keyspace to release resources
* Handle connect error
* Ignore drop keyspace errors
* Create pathDB with Cassandra client session
* Drop keyspace before session closed
* Measure pathDB operations (#1414)
* Adapt to path-mapped package changes (#1417)
* Add confset for subsys/cassandra (#1419)
* Adapt to path-mapped package changes
* Add confset for subsys/cassandra
* switch to using openshift api to control/follow image build (#1415) (#1418)
* switch to using openshift api to control/follow image build
* fix hook url var to my_bc in Jenkinsfile
* provide better feedback about bc selection process
* typo in stage end brace
* Set cassandraClient so-connection timeout and add close method (#1422)
* Add CassandraClient.close
* Add a debug log
* Extend cassandra cluster maxMem
* Back to standalone cassandra
* Put off close
* Try again by only close session
* use cassandraClient.close
* Fix TrackedContentEntryTest
* setConnectTimeoutMillis to 30 seconds
* Fix groovy.lang.MissingPropertyException: No such property: tarball_url (#1427)
* Fix npe in CassandraClient (#1431)
* Fix npe in CassandraClient
* Set socket options
* Addon path-mapped to provide direct listing/get/delete operations (#1436)
* Cherry fixes for Swagger and bump some versions (#1437)
* Fix the swagger-ui issue when upgraded to 3.23.x (#1435)
* Bump keycloakVersion from 6.0.0 to 7.0.1 (#1410)
Bumps `keycloakVersion` from 6.0.0 to 7.0.1.
Updates `keycloak-undertow-adapter` from 6.0.0 to 7.0.1
Updates `keycloak-core` from 6.0.0 to 7.0.1
- [Release notes](https://github.com/keycloak/keycloak/releases)
- [Commits](https://github.com/keycloak/keycloak/compare/6.0.0...7.0.1)
Updates `keycloak-adapter-core` from 6.0.0 to 7.0.1
- [Release notes](https://github.com/keycloak/keycloak/releases)
- [Commits](https://github.com/keycloak/keycloak/compare/6.0.0...7.0.1)
Signed-off-by: dependabot[bot] <support@github.com>
* Add query param limit to pathmapresource.list (#1439)
* Filter listing result by query param type (file,dir,all) (#1441)
* Filter listing result by query param type (file,dir,all)
* Update the filter
* Remove useless import
* Update
* Fix null fileType
* Merge master to path-mapped feature branch (#1459)
* Reuse the existing repo as implied one should add it into group
* NOS-2099:
Added information from validating RemoteRepositories inside RemoteRepository metadata object.
Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service.
* Test
* Filter the existing repos matching the conditions
* Enhance the condition for comparing the repo
* Deprecated listContents in client module as it is not working now
* Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana
* Tweak the condition and merge the keys of metadata "implied_by_stores"
* Changes Required
* only do deploy when we're doing an image build, and add timestamps to Folo tracking records
* Adding unit tests for version change in TrackedContentEntry
* Bug fix
* InternalFeature isn't null value
* DefaultStoreValidatorTest
* Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser
* Bump swagger-ui from 2.2.10 to 3.23.11 in /uis/layover
Bumps [swagger-ui](https://github.com/swagger-api/swagger-ui) from 2.2.10 to 3.23.11.
- [Release notes](https://github.com/swagger-api/swagger-ui/releases)
- [Commits](https://github.com/swagger-api/swagger-ui/compare/v2.2.10...v3.23.11)
Signed-off-by: dependabot[bot] <support@github.com>
* turn off deflate encoding support until we understand why it can corrupt downloaded archives
* Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques… (#1393)
* Reuse the existing repo as implied one should add it into group
* NOS-2099:
Added information from validating RemoteRepositories inside RemoteRepository metadata object.
Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service.
* Test
* Added MD5 and SHA1 digested hashes inside header for HTTP HEAD requests for browsing content - NOS-2048.Also i have rewrite @HEAD method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception.
* Filter the existing repos matching the conditions
* Enhance the condition for comparing the repo
* Fix Bad Request 400 error when uploading snapshot maven-metadata (#1389)
* Allow disable of content index, and add prometheus metrics adapter
Adding Dropwizard Metrics adapter to publish metrics to Prometheus via servlet, since our perf-test instance can't deploy GraphiteDB to a pod. This gets us ready for Prometheus anyway.
We may not need content indexing anymore in our particular deployment environment, and it might actually be hurting performance for our use case. Adding an option to disable it via config.
* add managed dep on indy-subsys-metrics-prometheus
* Fix tests broken by default-disabled content index, and improve some logger names
* Deprecated listContents in client module as it is not working now
* Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana
* Tweak the condition and merge the keys of metadata "implied_by_stores"
* Added Tests for checking http HEAD requests on '/api/browse/{packageType}/{type}/{name}/{path (.*)}'
* Changes Requested
* Changes Required
* only do deploy when we're doing an image build, and add timestamps to Folo tracking records
* Adding unit tests for version change in TrackedContentEntry
* Bug fix
* InternalFeature isn't null value
* DefaultStoreValidatorTest
* Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser
* Added Digesting Algos for Repository contents inside 'ResponseHelper' class
* Added Digesting Algos for Repository contents inside 'ResponseHelper' class
* Path not directory
* Content Digest if path doesn't end with '/'
* Added headers in ContentAcessHandler
* switch to using openshift api to control/follow image build (#1415)
* switch to using openshift api to control/follow image build
* fix hook url var to my_bc in Jenkinsfile
* provide better feedback about bc selection process
* typo in stage end brace
* Start path map storage migrator (#1416)
* Start path map storage migrator
* Fix deps and remove first slash of store path
* Bump jacksonVersion from 2.9.10 to 2.10.1
Bumps `jacksonVersion` from 2.9.10 to 2.10.1.
Updates `jackson-core` from 2.9.10 to 2.10.1
- [Release notes](https://github.com/FasterXML/jackson-core/releases)
- [Commits](https://github.com/FasterXML/jackson-core/compare/jackson-core-2.9.10...jackson-core-2.10.1)
Updates `jackson-annotations` from 2.9.10 to 2.10.1
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)
Updates `jackson-databind` from 2.9.10 to 2.10.1
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)
Updates `jackson-jaxrs-json-provider` from 2.9.10 to 2.10.1
Signed-off-by: dependabot[bot] <support@github.com>
* Add dedupe check support for pathmap migrator
* Use move instead of delete for processed path files in todo folder
* Add reload processed count for next migrate from failure (#1423)
* Fix the path mapped migrator checksum issue (#1424)
* Fix TrackedContentEntryTest (#1425)
* Avoid one not needed traverse of first level package path
And concurrently run scan from package level
* Fix groovy.lang.MissingPropertyException: No such property: tarball_url (#1428)
* Add filter to filter out the files not need to migrate
* [migration-tool] shutdown pathDB after done (#1430)
* Remove pathmap-migrator (#1432)
Pathmap-migrator has been moved to a single repo here:
https://www.github.com/Commonjava/pathmap-migrator
* Remove left pathmap-migrator class
* Fix the swagger-ui issue when upgraded to 3.23.x (#1435)
* Bump keycloakVersion from 6.0.0 to 7.0.1 (#1410)
Bumps `keycloakVersion` from 6.0.0 to 7.0.1.
Updates `keycloak-undertow-adapter` from 6.0.0 to 7.0.1
Updates `keycloak-core` from 6.0.0 to 7.0.1
- [Release notes](https://github.com/keycloak/keycloak/releases)
- [Commits](https://github.com/keycloak/keycloak/compare/6.0.0...7.0.1)
Updates `keycloak-adapter-core` from 6.0.0 to 7.0.1
- [Release notes](https://github.com/keycloak/keycloak/releases)
- [Commits](https://github.com/keycloak/keycloak/compare/6.0.0...7.0.1)
Signed-off-by: dependabot[bot] <support@github.com>
* NOS-2167: Handling bug issue in StoreValidator where process hang if remote repository is not valid (#1438)
* provide better feedback about bc selection process
* typo in stage end brace
* record and pass on data tarball to image build in addition to skinny tarball
* remove merge error; duplicate else clause
* Remove two dup lines
* adding trace logging to figure out why raw timings aren't being metered (#1445)
* Accumulate timing info for different types of calls in a given user request, then report these in the REST end log (#1446)
* skip all tests during deployment (they run earlier) (#1447)
* Fix cumulative timing measurement and add accumulation for non-intercepted metrics (#1448)
* Add creation time for all repos
Seems there is no creation time for all of our current repos. I think
it is useful for tracking, especially for the auto-created repos like
koji-proxy and implied repos.
BTW, I'm also thinking if we also need to add the "last_update_time"
for all repos.
* Level down metricsManager log for accumulating to trace
* Make repo create time as string for ser/de-ser consideration
* Switch to calculating cumulative elapsed milliseconds instead of seconds (#1451)
* add license headers and bump to release of galley
* [maven-release-plugin] prepare release indy-parent-1.9.6
* [maven-release-plugin] prepare for next development iteration
* Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager (#1455)
* Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager
Previously, storing ArtifactStore definitions as JSON requires us to serialize/deserialize
(render and parse) constantly, every time we access a repository definition. We can access
dozens or even thousands of these in a request, especially in an upload request or a
promotion request.
Looking at our current production deployment, we see that upload latency is up around 7 seconds
at times, with about 6.98 seconds of that being consumed in the process of working with the
store-definition cache.
This change implements Externalizable for all ArtifactStore derivatives, StoreKey, and related
classes in the core model of Indy. It adds some testing to ensure these can be read / written
appropriately using readExternal / writeExternal methods, and verifies that round-trip data is
preserved.
Then, it switches the cache used by InfinispanStoreDataManager to use Cache<StoreKey, ArtifactStore>
instead of Cache<StoreKey, String>, and relies on Infinispan itself to use the Externalizable
implementations to serialize data when it is persisted to the backing database.
Along with this comes a need to migrate existing JSON data to the new binary format. To handle
this, I've modified the InfinispanStoreDataMigrationAction class to look for a cache named
'store-data-v1' and if it finds it, to migrate the old-style store definitions from JSON to
the new cache, managed by InfinispanStoreDataManager. It will NOT use the really old filesystem
storage location if it finds this legacy cache.
* make no-arg constructors public and correct usage of Object[In/Out]putStream
* Add mixed-lot serialization for ArtifactStore, and read all by ArtifactStore base class
This will test that polymorphism works in the object stream.
* Cumulative call count in logging MDC and metering ratio (#1457)
* remove unused serialization code
* implement cumulative counts for meters, and add support for metering ratios to cut down on metering latency
* Using store-data-v1 for new object-based store-data cache (#1456)
I found a problem for migration process. When we use db-based
persistence for store cache, we can not control the table name for which
store use by applying a simple "table-name" like attr in ispn xml
because it does not support this. The table name is applied by its cache
name directly. So we need to use a new cache name for the new obj-based
store, ane use the legacy "store-data" cache for migration purpose.
* change the current store-data cache name to 'store-data-v2' to imply that 'store-data' is v1 (#1458)
* Merge master to pathmap (1.9.8-SNAPSHOT) (#1464)
* Reuse the existing repo as implied one should add it into group
* NOS-2099:
Added information from validating RemoteRepositories inside RemoteRepository metadata object.
Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service.
* Test
* Filter the existing repos matching the conditions
* Enhance the condition for comparing the repo
* Deprecated listContents in client module as it is not working now
* Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana
* Tweak the condition and merge the keys of metadata "implied_by_stores"
* Changes Required
* only do deploy when we're doing an image build, and add timestamps to Folo tracking records
* Adding unit tests for version change in TrackedContentEntry
* Bug fix
* InternalFeature isn't null value
* DefaultStoreValidatorTest
* Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser
* Bump swagger-ui from 2.2.10 to 3.23.11 in /uis/layover
Bumps [swagger-ui](https://github.com/swagger-api/swagger-ui) from 2.2.10 to 3.23.11.
- [Release notes](https://github.com/swagger-api/swagger-ui/releases)
- [Commits](https://github.com/swagger-api/swagger-ui/compare/v2.2.10...v3.23.11)
Signed-off-by: dependabot[bot] <support@github.com>
* turn off deflate encoding support until we understand why it can corrupt downloaded archives
* Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques… (#1393)
* Reuse the existing repo as implied one should add it into group
* NOS-2099:
Added information from validating RemoteRepositories inside RemoteRepository metadata object.
Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service.
* Test
* Added MD5 and SHA1 digested hashes inside header for HTTP HEAD requests for browsing content - NOS-2048.Also i have rewrite @HEAD method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception.
* Filter the existing repos matching the conditions
* Enhance the condition for comparing the repo
* Fix Bad Request 400 error when uploading snapshot maven-metadata (#1389)
* Allow disable of content index, and add prometheus metrics adapter
Adding Dropwizard Metrics adapter to publish metrics to Prometheus via servlet, since our perf-test instance can't deploy GraphiteDB to a pod. This gets us ready for Prometheus anyway.
We may not need content indexing anymore in our particular deployment environment, and it might actually be hurting performance for our use case. Adding an option to disable it via config.
* add managed dep on indy-subsys-metrics-prometheus
* Fix tests broken by default-disabled content index, and improve some logger names
* Deprecated listContents in client module as it is not working now
* Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana
* Tweak the condition and merge the keys of metadata "implied_by_stores"
* Added Tests for checking http HEAD requests on '/api/browse/{packageType}/{type}/{name}/{path (.*)}'
* Changes Requested
* Changes Required
* only do deploy when we're doing an image build, and add timestamps to Folo tracking records
* Adding unit tests for version change in TrackedContentEntry
* Bug fix
* InternalFeature isn't null value
* DefaultStoreValidatorTest
* Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser
* Added Digesting Algos for Repository contents inside 'ResponseHelper' class
* Added Digesting Algos for Repository contents inside 'ResponseHelper' class
* Path not directory
* Content Digest if path doesn't end with '/'
* Added headers in ContentAcessHandler
* switch to using openshift api to control/follow image build (#1415)
* switch to using openshift api to control/follow image build
* fix hook url var to my_bc in Jenkinsfile
* provide better feedback about bc selection process
* typo in stage end brace
* Start path map storage migrator (#1416)
* Start path map storage migrator
* Fix deps and remove first slash of store path
* Bump jacksonVersion from 2.9.10 to 2.10.1
Bumps `jacksonVersion` from 2.9.10 to 2.10.1.
Updates `jackson-core` from 2.9.10 to 2.10.1
- [Release notes](https://github.com/FasterXML/jackson-core/releases)
- [Commits](https://github.com/FasterXML/jackson-core/compare/jackson-core-2.9.10...jackson-core-2.10.1)
Updates `jackson-annotations` from 2.9.10 to 2.10.1
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)
Updates `jackson-databind` from 2.9.10 to 2.10.1
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)
Updates `jackson-jaxrs-json-provider` from 2.9.10 to 2.10.1
Signed-off-by: dependabot[bot] <support@github.com>
* Add dedupe check support for pathmap migrator
* Use move instead of delete for processed path files in todo folder
* Add reload processed count for next migrate from failure (#1423)
* Fix the path mapped migrator checksum issue (#1424)
* Fix TrackedContentEntryTest (#1425)
* Avoid one not needed traverse of first level package path
And concurrently run scan from package level
* Fix groovy.lang.MissingPropertyException: No such property: tarball_url (#1428)
* Add filter to filter out the files not need to migrate
* [migration-tool] shutdown pathDB after done (#1430)
* Remove pathmap-migrator (#1432)
Pathmap-migrator has been moved to a single repo here:
https://www.github.com/Commonjava/pathmap-migrator
* Remove left pathmap-migrator class
* Fix the swagger-ui issue when upgraded to 3.23.x (#1435)
* Bump keycloakVersion from 6.0.0 to 7.0.1 (#1410)
Bumps `keycloakVersion` from 6.0.0 to 7.0.1.
Updates `keycloak-undertow-adapter` from 6.0.0 to 7.0.1
Updates `keycloak-core` from 6.0.0 to 7.0.1
- [Release notes](https://github.com/keycloak/keycloak/releases)
- [Commits](https://github.com/keycloak/keycloak/compare/6.0.0...7.0.1)
Updates `keycloak-adapter-core` from 6.0.0 to 7.0.1
- [Release notes](https://github.com/keycloak/keycloak/releases)
- [Commits](https://github.com/keycloak/keycloak/compare/6.0.0...7.0.1)
Signed-off-by: dependabot[bot] <support@github.com>
* NOS-2167: Handling bug issue in StoreValidator where process hang if remote repository is not valid (#1438)
* provide better feedback about bc selection process
* typo in stage end brace
* record and pass on data tarball to image build in addition to skinny tarball
* remove merge error; duplicate else clause
* Remove two dup lines
* adding trace logging to figure out why raw timings aren't being metered (#1445)
* Accumulate timing info for different types of calls in a given user request, then report these in the REST end log (#1446)
* skip all tests during deployment (they run earlier) (#1447)
* Fix cumulative timing measurement and add accumulation for non-intercepted metrics (#1448)
* Add creation time for all repos
Seems there is no creation time for all of our current repos. I think
it is useful for tracking, especially for the auto-created repos like
koji-proxy and implied repos.
BTW, I'm also thinking if we also need to add the "last_update_time"
for all repos.
* Level down metricsManager log for accumulating to trace
* Make repo create time as string for ser/de-ser consideration
* Switch to calculating cumulative elapsed milliseconds instead of seconds (#1451)
* add license headers and bump to release of galley
* [maven-release-plugin] prepare release indy-parent-1.9.6
* [maven-release-plugin] prepare for next development iteration
* Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager (#1455)
* Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager
Previously, storing ArtifactStore definitions as JSON requires us to serialize/deserialize
(render and parse) constantly, every time we access a repository definition. We can access
dozens or even thousands of these in a request, especially in an upload request or a
promotion request.
Looking at our current production deployment, we see that upload latency is up around 7 seconds
at times, with about 6.98 seconds of that being consumed in the process of working with the
store-definition cache.
This change implements Externalizable for all ArtifactStore derivatives, StoreKey, and related
classes in the core model of Indy. It adds some testing to ensure these can be read / written
appropriately using readExternal / writeExternal methods, and verifies that round-trip data is
preserved.
Then, it switches the cache used by InfinispanStoreDataManager to use Cache<StoreKey, ArtifactStore>
instead of Cache<StoreKey, String>, and relies on Infinispan itself to use the Externalizable
implementations to serialize data when it is persisted to the backing database.
Along with this comes a need to migrate existing JSON data to the new binary format. To handle
this, I've modified the InfinispanStoreDataMigrationAction class to look for a cache named
'store-data-v1' and if it finds it, to migrate the old-style store definitions from JSON to
the new cache, managed by InfinispanStoreDataManager. It will NOT use the really old filesystem
storage location if it finds this legacy cache.
* make no-arg constructors public and correct usage of Object[In/Out]putStream
* Add mixed-lot serialization for ArtifactStore, and read all by ArtifactStore base class
This will test that polymorphism works in the object stream.
* Cumulative call count in logging MDC and metering ratio (#1457)
* remove unused serialization code
* implement cumulative counts for meters, and add support for metering ratios to cut down on metering latency
* Using store-data-v1 for new object-based store-data cache (#1456)
I found a problem for migration process. When we use db-based
persistence for store cache, we can not control the table name for which
store use by applying a simple "table-name" like attr in ispn xml
because it does not support this. The table name is applied by its cache
name directly. So we need to use a new cache name for the new obj-based
store, ane use the legacy "store-data" cache for migration purpose.
* change the current store-data cache name to 'store-data-v2' to imply that 'store-data' is v1 (#1458)
* Drill down metrics for store.execute (#1460)
* Remove old secrets loading code in indy.sh (#1453)
* Rewrite store-query methods that use stream() and intercept/wrap store-query for metrics (#1461)
* Rewrite store-query methods that use stream() and intercept/wrap store-query for metrics
We cannot use the @Measure annotation in ArtifactStoreQuery, since that class is instantiated a variable number of
calls for each user request, and instances hold state supporting the fluent API. The StoreDataManager decorator
wraps ArtifactStoreQuery results from the query() method in a query class that measures query calls, delegating the
actual query to the instance that comes from the underlying StoreDataManager delegate.
StoreDataManager stream / getAll methods rely on having in-memory access to ALL of the ArtifactStore instances,
and also on having a small enough instance count to make iterating through all of them feasible. In our production
environment, both of these assumptions are violated. We need a better way to access this information, but to start
I've refactored DefaultArtifactStoreQuery methods that use streams and getAll in the StoreDataManager so they won't
do that.
In the case of retrieving the stores that are in the membership tree of a particular group, I've rewritten that to
avoid recursion.
In addition, I've added a keyStream() method to ArtifactStoreQuery and a streamArtifactStoreKeys() method to
StoreDataManager. This SHOULD be something that we can optimize, ensuring all keys are available in-memory or easy
to deserialize with minimum overhead. We can do a lot of filtering based on the key alone, then additional
filtering can be applied with the matching stores, which would be retrieved directly by key from the filtered
keys in these calls.
* remove duplicate dependencies in 2 poms
* Separate metered-db stuff to break dependency cycle, and fix a few compilation / unit test problems
* Add license headers
* [maven-release-plugin] prepare release indy-parent-1.9.7
* [maven-release-plugin] prepare for next development iteration
* Add tck for ispn store data manager
And following:
* One more tck case for complex group hierarchy for affected by
testing
* Removed commented code in infinispan data store manager for pretty
* FIX: NPE happened during getAffectedBy
Seems the new refactored code brings some new way to get affected
groups, but it missed some NPE check which caused 500 error.
* update license headers in new classes
* [maven-release-plugin] prepare release indy-parent-1.9.7.1
* [maven-release-plugin] prepare for next development iteration
* Update snapshot version to 1.9.8
* Fix mistakes
* Add path-mapped-storage dep version to indy pom (#1466)
* Make cassandra-maven-plugin run only in ci mode
* Merge master to integration pathmap (#1467)
* Merge master to pathmap branch (#1501)
* Reuse the existing repo as implied one should add it into group
* NOS-2099:
Added information from validating RemoteRepositories inside RemoteRepository metadata object.
Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service.
* Test
* Filter the existing repos matching the conditions
* Enhance the condition for comparing the repo
* Deprecated listContents in client module as it is not working now
* Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana
* Tweak the condition and merge the keys of metadata "implied_by_stores"
* Changes Required
* only do deploy when we're doing an image build, and add timestamps to Folo tracking records
* Adding unit tests for version change in TrackedContentEntry
* Bug fix
* InternalFeature isn't null value
* DefaultStoreValidatorTest
* Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser
* Bump swagger-ui from 2.2.10 to 3.23.11 in /uis/layover
Bumps [swagger-ui](https://github.com/swagger-api/swagger-ui) from 2.2.10 to 3.23.11.
- [Release notes](https://github.com/swagger-api/swagger-ui/releases)
- [Commits](https://github.com/swagger-api/swagger-ui/compare/v2.2.10...v3.23.11)
Signed-off-by: dependabot[bot] <support@github.com>
* turn off deflate encoding support until we understand why it can corrupt downloaded archives
* Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques… (#1393)
* Reuse the existing repo as implied one should add it into group
* NOS-2099:
Added information from validating RemoteRepositories inside RemoteRepository metadata object.
Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service.
* Test
* Added MD5 and SHA1 digested hashes inside header for HTTP HEAD requests for browsing content - NOS-2048.Also i have rewrite @HEAD method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception.
* Filter the existing repos matching the conditions
* Enhance the condition for comparing the repo
* Fix Bad Request 400 error when uploading snapshot maven-metadata (#1389)
* Allow disable of content index, and add prometheus metrics adapter
Adding Dropwizard Metrics adapter to publish metrics to Prometheus via servlet, since our perf-test instance can't deploy GraphiteDB to a pod. This gets us ready for Prometheus anyway.
We may not need content indexing anymore in our particular deployment environment, and it might actually be hurting performance for our use case. Adding an option to disable it via config.
* add managed dep on indy-subsys-metrics-prometheus
* Fix tests broken by default-disabled content index, and improve some logger names
* Deprecated listContents in client module as it is not working now
* Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana
* Tweak the condition and merge the keys of metadata "implied_by_stores"
* Added Tests for checking http HEAD requests on '/api/browse/{packageType}/{type}/{name}/{path (.*)}'
* Changes Requested
* Changes Required
* only do deploy when we're doing an image build, and add timestamps to Folo tracking records
* Adding unit tests for version change in TrackedContentEntry
* Bug fix
* InternalFeature isn't null value
* DefaultStoreValidatorTest
* Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser
* Added Digesting Algos for Repository contents inside 'ResponseHelper' class
* Added Digesting Algos for Repository contents inside 'ResponseHelper' class
* Path not directory
* Content Digest if path doesn't end with '/'
* Added headers in ContentAcessHandler
* switch to using openshift api to control/follow image build (#1415)
* switch to using openshift api to control/follow image build
* fix hook url var to my_bc in Jenkinsfile
* provide better feedback about bc selection process
* typo in stage end brace
* Start path map storage migrator (#1416)
* Start path map storage migrator
* Fix deps and remove first slash of store path
* Bump jacksonVersion from 2.9.10 to 2.10.1
Bumps `jacksonVersion` from 2.9.10 to 2.10.1.
Updates `jackson-core` from 2.9.10 to 2.10.1
- [Release notes](https://github.com/FasterXML/jackson-core/releases)
- [Commits](https://github.com/FasterXML/jackson-core/compare/jackson-core-2.9.10...jackson-core-2.10.1)
Updates `jackson-annotations` from 2.9.10 to 2.10.1
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)
Updates `jackson-databind` from 2.9.10 to 2.10.1
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)
Updates `jackson-jaxrs-json-provider` from 2.9.10 to 2.10.1
Signed-off-by: dependabot[bot] <support@github.com>
* Add dedupe check support for pathmap migrator
* Use move instead of delete for processed path files in todo folder
* Add reload processed count for next migrate from failure (#1423)
* Fix the path mapped migrator checksum issue (#1424)
* Fix TrackedContentEntryTest (#1425)
* Avoid one not needed traverse of first level package path
And concurrently run scan from package level
* Fix groovy.lang.MissingPropertyException: No such property: tarball_url (#1428)
* Add filter to filter out the files not need to migrate
* [migration-tool] shutdown pathDB after done (#1430)
* Remove pathmap-migrator (#1432)
Pathmap-migrator has been moved to a single repo here:
https://www.github.com/Commonjava/pathmap-migrator
* Remove left pathmap-migrator class
* Fix the swagger-ui issue when upgraded to 3.23.x (#1435)
* Bump keycloakVersion from 6.0.0 to 7.0.1 (#1410)
Bumps `keycloakVersion` from 6.0.0 to 7.0.1.
Updates `keycloak-undertow-adapter` from 6.0.0 to 7.0.1
Updates `keycloak-core` from 6.0.0 to 7.0.1
- [Release notes](https://github.com/keycloak/keycloak/releases)
- [Commits](https://github.com/keycloak/keycloak/compare/6.0.0...7.0.1)
Updates `keycloak-adapter-core` from 6.0.0 to 7.0.1
- [Release notes](https://github.com/keycloak/keycloak/releases)
- [Commits](https://github.com/keycloak/keycloak/compare/6.0.0...7.0.1)
Signed-off-by: dependabot[bot] <support@github.com>
* NOS-2167: Handling bug issue in StoreValidator where process hang if remote repository is not valid (#1438)
* provide better feedback about bc selection process
* typo in stage end brace
* record and pass on data tarball to image build in addition to skinny tarball
* remove merge error; duplicate else clause
* Remove two dup lines
* adding trace logging to figure out why raw timings aren't being metered (#1445)
* Accumulate timing info for different types of calls in a given user request, then report these in the REST end log (#1446)
* skip all tests during deployment (they run earlier) (#1447)
* Fix cumulative timing measurement and add accumulation for non-intercepted metrics (#1448)
* Add creation time for all repos
Seems there is no creation time for all of our current repos. I think
it is useful for tracking, especially for the auto-created repos like
koji-proxy and implied repos.
BTW, I'm also thinking if we also need to add the "last_update_time"
for all repos.
* Level down metricsManager log for accumulating to trace
* Make repo create time as string for ser/de-ser consideration
* Switch to calculating cumulative elapsed milliseconds instead of seconds (#1451)
* add license headers and bump to release of galley
* [maven-release-plugin] prepare release indy-parent-1.9.6
* [maven-release-plugin] prepare for next development iteration
* Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager (#1455)
* Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager
Previously, storing ArtifactStore definitions as JSON requires us to serialize/deserialize
(render and parse) constantly, every time we access a repository definition. We can access
dozens or even thousands of these in a request, especially in an upload request or a
promotion request.
Looking at our current production deployment, we see that upload latency is up around 7 seconds
at times, with about 6.98 seconds of that being consumed in the process of working with the
store-definition cache.
This change implements Externalizable for all ArtifactStore derivatives, StoreKey, and related
classes in the core model of Indy. It adds some testing to ensure these can be read / written
appropriately using readExternal / writeExternal methods, and verifies that round-trip data is
preserved.
Then, it switches the cache used by InfinispanStoreDataManager to use Cache<StoreKey, ArtifactStore>
instead of Cache<StoreKey, String>, and relies on Infinispan itself to use the Externalizable
implementations to serialize data when it is persisted to the backing database.
Along with this comes a need to migrate existing JSON data to the new binary format. To handle
this, I've modified the InfinispanStoreDataMigrationAction class to look for a cache named
'store-data-v1' and if it finds it, to migrate the old-style store definitions from JSON to
the new cache, managed by InfinispanStoreDataManager. It will NOT use the really old filesystem
storage location if it finds this legacy cache.
* make no-arg constructors public and correct usage of Object[In/Out]putStream
* Add mixed-lot serialization for ArtifactStore, and read all by ArtifactStore base class
This will test that polymorphism works in the object stream.
* Cumulative call count in logging MDC and metering ratio (#1457)
* remove unused serialization code
* implement cumulative counts for meters, and add support for metering ratios to cut down on metering latency
* Using store-data-v1 for new object-based store-data cache (#1456)
I found a problem for migration process. When we use db-based
persistence for store cache, we can not control the table name for which
store use by applying a simple "table-name" like attr in ispn xml
because it does not support this. The table name is applied by its cache
name directly. So we need to use a new cache name for the new obj-based
store, ane use the legacy "store-data" cache for migration purpose.
* change the current store-data cache name to 'store-data-v2' to imply that 'store-data' is v1 (#1458)
* Drill down metrics for store.execute (#1460)
* Remove old secrets loading code in indy.sh (#1453)
* Rewrite store-query methods that use stream() and intercept/wrap store-query for metrics (#1461)
* Rewrite store-query methods that use stream() and intercept/wrap store-query for metrics
We cannot use the @Measure annotation in ArtifactStoreQuery, since that class is instantiated a variable number of
calls for each user request, and instances hold state supporting the fluent API. The StoreDataManager decorator
wraps ArtifactStoreQuery results from the query() method in a query class that measures query calls, delegating the
actual query to the instance that comes from the underlying StoreDataManager delegate.
StoreDataManager stream / getAll methods rely on having in-memory access to ALL of the ArtifactStore instances,
and also on having a small enough instance count to make iterating through all of them feasible. In our production
environment, both of these assumptions are violated. We need a better way to access this information, but to start
I've refactored DefaultArtifactStoreQuery methods that use streams and getAll in the StoreDataManager so they won't
do that.
In the case of retrieving the stores that are in the membership tree of a particular group, I've rewritten that to
avoid recursion.
In addition, I've added a keyStream() method to ArtifactStoreQuery and a streamArtifactStoreKeys() method to
StoreDataManager. This SHOULD be something that we can optimize, ensuring all keys are available in-memory or easy
to deserialize with minimum overhead. We can do a lot of filtering based on the key alone, then additional
filtering can be applied with the matching stores, which would be retrieved directly by key from the filtered
keys in these calls.
* remove duplicate dependencies in 2 poms
* Separate metered-db stuff to break dependency cycle, and fix a few compilation / unit test problems
* Add license headers
* [maven-release-plugin] prepare release indy-parent-1.9.7
* [maven-release-plugin] prepare for next development iteration
* Add tck for ispn store data manager
And following:
* One more tck case for complex group hierarchy for affected by
testing
* Removed commented code in infinispan data store manager for pretty
* FIX: NPE happened during getAffectedBy
Seems the new refactored code brings some new way to get affected
groups, but it missed some NPE check which caused 500 error.
* update license headers in new classes
* [maven-release-plugin] prepare release indy-parent-1.9.7.1
* [maven-release-plugin] prepare for next development iteration
* Remove unused jdk internal imports
* Add comment for new group mangement tck case
* Extract getStore operation from deep loop for affected by
This getArtifactStore resides in a deep loop, which will cause serious
performance problem if the loop size is very huge(Actually it is). So
need to extract it out from this loop.
* Async the group de-index during upload in content-index service
Recent performance problem we met for uploading & promotion is all
caused by the getAffectedBy operation, which is a time-consumed one for
a full scan on store data. After checking, I found that it is mainly
used for de-index or re-index content-index entries for the uploading
repo. I think this de-index will not be a block factor during the upload
or promotion process, which means we could make it async.
In this commit, I created a ISPN cache and used it like a queue, which
decouple the de-index opeartion from storing process in content-index to
an aync listener of ISPN listener.
Why not use a weft executor? Because the storing operation is high
frequent, which means it could be blocked again if the de-index
effeciency is much slower than the real physical storing operation. Use
this queue-liked way we can delegate all these to underlying ISPN and we
have a position to set the queue size from ISPN config.
* Revert "Async the group de-index during upload in content-index service"
This reverts commit 0b3ef58f054dab2f336d172ba0e839429b339b5e.
With load-testing, found that there are a bunch of ispn lock acquiring
timeout error, so that means the getAffectedBy operation is much
slower than file transferring itself, and cause many unhandled
de-index entries in that cache. We need to think about another
solution to switch to async mode.
* Async the group de-index during upload in content-index service
Recent performance problem we met for uploading & promotion is all
caused by the getAffectedBy operation, which is a time-consumed one for
a full scan on store data. After checking, I found that it is mainly
used for de-index or re-index content-index entries for the uploading
repo. I think this de-index will not be a block factor during the upload
or promotion process, which means we could make it async.
In this commit, I switched back to use a normal executor service to
async the de-index of the affected group. One potential problem for this
solution is the executor queue size, which could cause OOM if there are
bunch of uploading happened in a short time period. We need to monitor
if this can happen.
* Code cleanup: remove dupe and unused deps (#1474)
Purpose to reduce indy launcher tarball size. Did following:
* Removed all duped junit, harmcrest and cdi delaration
* Removed quartz deps as it is not used now
* Removed h2 as it is not used now
* don't build cache migrator or record extractor in main build. (#1478)
These don't need to be on the main release cycle, and it seems there's some recursion
problem in the maven-assembly-plugin with one of them that's screwing up Indy builds.
* Thread-off two nfc cleaning ops which refers getAffectedBy (#1477)
* Thread-off getAffectedBy ops for nfc cleaning in ContentManager
* Thread-off getAffectedBy ops for nfc cleaning in promote validation
* register NFC cache listener in DefaultContentIndexManager (#1479)
* Limit "affected-by" with full scan opeartion in package and type based set (#1471)
Add a new Cache which is <package, Map<type, Set<StoreKey>> to store
all Store keys based on package and type, so for affected group get
method, we can avoid the full scan on all repos to get groups, but just
simplified to two map get ops.
* Fix NPE when store is null in case of being deleted
* Thread-off getAffectedBy ops in NFCContentListener
* Extract getGroupsAffectedBy from Query to StoreDataManager
* Add reverse map for affected stores by some constituents change
* Limit "affected-by" with full scan opeartion in package and type based set (#1471)
Add a new Cache which is <package, Map<type, Set<StoreKey>> to store
all Store keys based on package and type, so for affected group get
method, we can avoid the full scan on all repos to get groups, but just
simplified to two map get ops.
* Simplify affected-by cache
We won't completely flatten out the affected-by cache by resolving groups affected by groups affected by a given store. Instead, we leave it as a simple reverse map of the group's constituents, which will give us direct keys to lookup in order to iterate and calculated these n-level affected relationships. It's still much better than scanning all groups (which in the past meant scanning all repos). At the same time, it will avoid any unforseen problems with group membership changes that don't get propagated via the flattening process. It should be easier to maintain.
* populate the affected-by cache on startup if the affected-by cache is clear
This should work if the affected-by cache is kept in memory, or for the first run when we use a database to
store it. We're lining this up with the by-package-type cache as a single-node fix, where we will still need
a more scalable solution from a proper database / data design approach before we're ready for multi-node usage.
* clean up affected-by iteration and consolidate iteration in default store query
I added a null check and disabled-group check to the iteration for finding groups affected by a particular store.
The rest of these are just adjusting away from streams and toward intermediary Set instances...things like that.
* fix merge error in StoreDataManager
* fix remaining compilation errors left from merging
* fix NPE when ssl validation repo is null due to some error
* Fix potential NPE by using PackageMetadataMerger as a component, as intended.
* Update copyright year range to include 2020, reformat license headers (#1484)
* fix license headers
* Update project copyright year range to include 2020 and reformat license headers
* [maven-release-plugin] prepare release indy-parent-1.9.8
* [maven-release-plugin] prepare for next development iteration
* Fix some code smell for PromotionManager like potential NPE
Conflicts:
api/src/main/java/org/commonjava/indy/data/StoreDataManager.java
* Some code simplify
Conflicts:
db/common/src/main/java/org/commonjava/indy/db/common/AbstractStoreDataManager.java
* Code optimization - MavenMetadataGenerator
* Simplify lambda
* Remove useless null check
* Use try resource instead
* Charset standardization
* Add some trace log for metadata clearing of pom update (#1487)
* Fix init of affected-by reverse map, and add checks/error message if it's empty (#1488)
* Fix affected-by reversemap initialization in lifecycle startup
* Add logging to note when the affected-by cache is altered, and another error log message if it's empty
* Refactor InfinispanStoreDataManager.init() to startup actions, add some logging
The init() method runs before any migration actions run, which means migrated stores won't be stored in the byPkg cache. The old initializer for the affected-by map has a similar problem, so I removed it in favor of the new startup action. The byPkg init was moved out to a startup action too for this reason.
Finally, I've added a log message to show which groups were found to be affected by a specific set of input keys. This will give us better information about whether the affected-by map is being used or not.
* adding another ftest that initializes store data at boot instead of at migration
* Support honeycomb
* Fix config
* Add HoneycombInterceptor
* Fix getchildspan
* Add HoneycombFilter
* Add pathInfo to fields
* Fix NPE in Honeycomb filter and add some fields to honeycomb spans
* Enable collection of more metrics (esp. I/O level) via Honeycomb (#1492)
This required refactoring the Galley timing function providers, using a new patch of Galley that
attempts to generalize the metrics interface, at least for I/O operations.
It also involved creating three new interceptors, which would handle methods that are designed to
wrap arbitrary pieces of code in metrics.
* return the Honeycomb trace span at the filter level, not the root span that is inside it. (#1493)
* add golden signals filter into IndyDeployment because filter ordering is important (#1494)
* fix cyclic dependency caused by moving GoldenSignalsFilter (#1495)
* fix cyclic dependency caused by moving GoldenSignalsFilter
* Adding beans.xml to allow CDI injection properly
* upgrade galley to release and fix license headers
* [maven-release-plugin] prepare release indy-parent-1.9.9
* [maven-release-plugin] prepare for next development iteration
* Honeycomb field-capture improvements, and span configuration (#1497)
* Fix the Honeycomb root span (which I broke previously)
* debug some honeycomb field-tracking problems
I had to break out the ThreadContext initialization into an outer servlet filter (specified first in the mapping order, since it's the outer one) to preserve fields until honeycomb can find it.
I've also added the ability to turn spans on/off in the configuration, using EITHER include OR exclude semantics (you can't use both).
* ignore metrics testing for now...test needs to be updated
* Consolidate all async getGroupAffectedBy ops in StoreDataManager
* New Relic telemetry support (#1498)
* Add New Relic telemetry support, and complete configuration of interceptors in beans.xml for Honeycomb
* check before sending batch, push field extraction to manager, use Dequeue instead of List
* Add New Relic telemetry support, and complete configuration of interceptors in beans.xml for Honeycomb
* check before sending batch, push field extraction to manager, use Dequeue instead of List
* Refine / fix new relic span tracking
* try to forget we ever used New Relic tracing, the UI is terrible
* Add new configuration approach for honeycomb and configurable sampling rates
Sampling rates can be specified by Golden Signals (SLI) function name, or by method name, or by class name + method name, etc.
* forgot to add configuration parameter for base sample rate
* fix compilation error
* Tie in threaded-off calls to Honeycomb trace by simulating jump to another service
* refactored thread contextualizer for honeycomb
I'm ignoring the client bound to other threads and just creating a new one from config when needed. This should work around NPEs in deeper / uncontrolled threads.
* Align version to 1.9.10-SNAPSHOT
* Update license head
* Fix merge issues
* Fix merge issues-2
* Fix 2 ftests where the bytestream is drained after the first use
* Revert "Fix 2 ftests where the bytestream is drained after the first use"
This reverts commit 81df08a2ab3a38d066b65be8b888bbf739c4c223.
Co-authored-by: Wenjie Guo <guowj12@gmail.com>
Co-authored-by: georgy <3187346+gorgija@users.noreply.github.com>
Co-authored-by: Gang Li <ligangty@gmail.com>
Co-authored-by: John Casey <jdcasey@commonjava.org>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: geored <47973913+geored@users.noreply.github.com>
* Fix merge issue
* Solve conflict
* Solve conflict
* Not delete read-only legacy files
* Use constants
Co-authored-by: Gang Li <ligangty@gmail.com>
Co-authored-by: Wenjie Guo <guowj12@gmail.com>
Co-authored-by: georgy <3187346+gorgija@users.noreply.github.com>
Co-authored-by: John Casey <jdcasey@commonjava.org>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: geored <47973913+geored@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…ts for browsing content - NOS-2048.Also i have rewrite @Head method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception.