Fix groovy.lang.MissingPropertyException: No such property: tarball_url by ruhan1 · Pull Request #1428 · Commonjava/indy

ruhan1 · 2019-11-18T03:12:38Z

The tarball_url is not a built-in variable but calculated in this script.

ruhan1 · 2019-11-18T03:13:46Z

I tested it in feature branch, and merge it to master directly. cc @jdcasey

@Head

* Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Bump swagger-ui from 2.2.10 to 3.23.11 in /uis/layover Bumps [swagger-ui](https://github.com/swagger-api/swagger-ui) from 2.2.10 to 3.23.11. - [Release notes](https://github.com/swagger-api/swagger-ui/releases) - [Commits](swagger-api/swagger-ui@v2.2.10...v3.23.11) Signed-off-by: dependabot[bot] <support@github.com> * turn off deflate encoding support until we understand why it can corrupt downloaded archives * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques… (#1393) * Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD requests for browsing content - NOS-2048.Also i have rewrite @Head method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception. * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Fix Bad Request 400 error when uploading snapshot maven-metadata (#1389) * Allow disable of content index, and add prometheus metrics adapter Adding Dropwizard Metrics adapter to publish metrics to Prometheus via servlet, since our perf-test instance can't deploy GraphiteDB to a pod. This gets us ready for Prometheus anyway. We may not need content indexing anymore in our particular deployment environment, and it might actually be hurting performance for our use case. Adding an option to disable it via config. * add managed dep on indy-subsys-metrics-prometheus * Fix tests broken by default-disabled content index, and improve some logger names * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Added Tests for checking http HEAD requests on '/api/browse/{packageType}/{type}/{name}/{path (.*)}' * Changes Requested * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Path not directory * Content Digest if path doesn't end with '/' * Added headers in ContentAcessHandler * switch to using openshift api to control/follow image build (#1415) * switch to using openshift api to control/follow image build * fix hook url var to my_bc in Jenkinsfile * provide better feedback about bc selection process * typo in stage end brace * Start path map storage migrator (#1416) * Start path map storage migrator * Fix deps and remove first slash of store path * Bump jacksonVersion from 2.9.10 to 2.10.1 Bumps `jacksonVersion` from 2.9.10 to 2.10.1. Updates `jackson-core` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson-core/releases) - [Commits](FasterXML/jackson-core@jackson-core-2.9.10...jackson-core-2.10.1) Updates `jackson-annotations` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) Updates `jackson-databind` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) Updates `jackson-jaxrs-json-provider` from 2.9.10 to 2.10.1 Signed-off-by: dependabot[bot] <support@github.com> * Add dedupe check support for pathmap migrator * Use move instead of delete for processed path files in todo folder * Add reload processed count for next migrate from failure (#1423) * Fix the path mapped migrator checksum issue (#1424) * Fix TrackedContentEntryTest (#1425) * Avoid one not needed traverse of first level package path And concurrently run scan from package level * Fix groovy.lang.MissingPropertyException: No such property: tarball_url (#1428) * Add filter to filter out the files not need to migrate * [migration-tool] shutdown pathDB after done (#1430) * Remove pathmap-migrator (#1432) Pathmap-migrator has been moved to a single repo here: https://www.github.com/Commonjava/pathmap-migrator * Remove left pathmap-migrator class * Fix the swagger-ui issue when upgraded to 3.23.x (#1435) * Bump keycloakVersion from 6.0.0 to 7.0.1 (#1410) Bumps `keycloakVersion` from 6.0.0 to 7.0.1. Updates `keycloak-undertow-adapter` from 6.0.0 to 7.0.1 Updates `keycloak-core` from 6.0.0 to 7.0.1 - [Release notes](https://github.com/keycloak/keycloak/releases) - [Commits](keycloak/keycloak@6.0.0...7.0.1) Updates `keycloak-adapter-core` from 6.0.0 to 7.0.1 - [Release notes](https://github.com/keycloak/keycloak/releases) - [Commits](keycloak/keycloak@6.0.0...7.0.1) Signed-off-by: dependabot[bot] <support@github.com> * NOS-2167: Handling bug issue in StoreValidator where process hang if remote repository is not valid (#1438) * provide better feedback about bc selection process * typo in stage end brace * record and pass on data tarball to image build in addition to skinny tarball * remove merge error; duplicate else clause * Remove two dup lines * adding trace logging to figure out why raw timings aren't being metered (#1445) * Accumulate timing info for different types of calls in a given user request, then report these in the REST end log (#1446) * skip all tests during deployment (they run earlier) (#1447) * Fix cumulative timing measurement and add accumulation for non-intercepted metrics (#1448) * Add creation time for all repos Seems there is no creation time for all of our current repos. I think it is useful for tracking, especially for the auto-created repos like koji-proxy and implied repos. BTW, I'm also thinking if we also need to add the "last_update_time" for all repos. * Level down metricsManager log for accumulating to trace * Make repo create time as string for ser/de-ser consideration * Switch to calculating cumulative elapsed milliseconds instead of seconds (#1451) * add license headers and bump to release of galley * [maven-release-plugin] prepare release indy-parent-1.9.6 * [maven-release-plugin] prepare for next development iteration * Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager (#1455) * Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager Previously, storing ArtifactStore definitions as JSON requires us to serialize/deserialize (render and parse) constantly, every time we access a repository definition. We can access dozens or even thousands of these in a request, especially in an upload request or a promotion request. Looking at our current production deployment, we see that upload latency is up around 7 seconds at times, with about 6.98 seconds of that being consumed in the process of working with the store-definition cache. This change implements Externalizable for all ArtifactStore derivatives, StoreKey, and related classes in the core model of Indy. It adds some testing to ensure these can be read / written appropriately using readExternal / writeExternal methods, and verifies that round-trip data is preserved. Then, it switches the cache used by InfinispanStoreDataManager to use Cache<StoreKey, ArtifactStore> instead of Cache<StoreKey, String>, and relies on Infinispan itself to use the Externalizable implementations to serialize data when it is persisted to the backing database. Along with this comes a need to migrate existing JSON data to the new binary format. To handle this, I've modified the InfinispanStoreDataMigrationAction class to look for a cache named 'store-data-v1' and if it finds it, to migrate the old-style store definitions from JSON to the new cache, managed by InfinispanStoreDataManager. It will NOT use the really old filesystem storage location if it finds this legacy cache. * make no-arg constructors public and correct usage of Object[In/Out]putStream * Add mixed-lot serialization for ArtifactStore, and read all by ArtifactStore base class This will test that polymorphism works in the object stream. * Cumulative call count in logging MDC and metering ratio (#1457) * remove unused serialization code * implement cumulative counts for meters, and add support for metering ratios to cut down on metering latency * Using store-data-v1 for new object-based store-data cache (#1456) I found a problem for migration process. When we use db-based persistence for store cache, we can not control the table name for which store use by applying a simple "table-name" like attr in ispn xml because it does not support this. The table name is applied by its cache name directly. So we need to use a new cache name for the new obj-based store, ane use the legacy "store-data" cache for migration purpose. * change the current store-data cache name to 'store-data-v2' to imply that 'store-data' is v1 (#1458)

@Head

* Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Bump swagger-ui from 2.2.10 to 3.23.11 in /uis/layover Bumps [swagger-ui](https://github.com/swagger-api/swagger-ui) from 2.2.10 to 3.23.11. - [Release notes](https://github.com/swagger-api/swagger-ui/releases) - [Commits](swagger-api/swagger-ui@v2.2.10...v3.23.11) Signed-off-by: dependabot[bot] <support@github.com> * turn off deflate encoding support until we understand why it can corrupt downloaded archives * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques… (#1393) * Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD requests for browsing content - NOS-2048.Also i have rewrite @Head method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception. * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Fix Bad Request 400 error when uploading snapshot maven-metadata (#1389) * Allow disable of content index, and add prometheus metrics adapter Adding Dropwizard Metrics adapter to publish metrics to Prometheus via servlet, since our perf-test instance can't deploy GraphiteDB to a pod. This gets us ready for Prometheus anyway. We may not need content indexing anymore in our particular deployment environment, and it might actually be hurting performance for our use case. Adding an option to disable it via config. * add managed dep on indy-subsys-metrics-prometheus * Fix tests broken by default-disabled content index, and improve some logger names * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Added Tests for checking http HEAD requests on '/api/browse/{packageType}/{type}/{name}/{path (.*)}' * Changes Requested * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Path not directory * Content Digest if path doesn't end with '/' * Added headers in ContentAcessHandler * switch to using openshift api to control/follow image build (#1415) * switch to using openshift api to control/follow image build * fix hook url var to my_bc in Jenkinsfile * provide better feedback about bc selection process * typo in stage end brace * Start path map storage migrator (#1416) * Start path map storage migrator * Fix deps and remove first slash of store path * Bump jacksonVersion from 2.9.10 to 2.10.1 Bumps `jacksonVersion` from 2.9.10 to 2.10.1. Updates `jackson-core` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson-core/releases) - [Commits](FasterXML/jackson-core@jackson-core-2.9.10...jackson-core-2.10.1) Updates `jackson-annotations` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) Updates `jackson-databind` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) Updates `jackson-jaxrs-json-provider` from 2.9.10 to 2.10.1 Signed-off-by: dependabot[bot] <support@github.com> * Add dedupe check support for pathmap migrator * Use move instead of delete for processed path files in todo folder * Add reload processed count for next migrate from failure (#1423) * Fix the path mapped migrator checksum issue (#1424) * Fix TrackedContentEntryTest (#1425) * Avoid one not needed traverse of first level package path And concurrently run scan from package level * Fix groovy.lang.MissingPropertyException: No such property: tarball_url (#1428) * Add filter to filter out the files not need to migrate * [migration-tool] shutdown pathDB after done (#1430) * Remove pathmap-migrator (#1432) Pathmap-migrator has been moved to a single repo here: https://www.github.com/Commonjava/pathmap-migrator * Remove left pathmap-migrator class * Fix the swagger-ui issue when upgraded to 3.23.x (#1435) * Bump keycloakVersion from 6.0.0 to 7.0.1 (#1410) Bumps `keycloakVersion` from 6.0.0 to 7.0.1. Updates `keycloak-undertow-adapter` from 6.0.0 to 7.0.1 Updates `keycloak-core` from 6.0.0 to 7.0.1 - [Release notes](https://github.com/keycloak/keycloak/releases) - [Commits](keycloak/keycloak@6.0.0...7.0.1) Updates `keycloak-adapter-core` from 6.0.0 to 7.0.1 - [Release notes](https://github.com/keycloak/keycloak/releases) - [Commits](keycloak/keycloak@6.0.0...7.0.1) Signed-off-by: dependabot[bot] <support@github.com> * NOS-2167: Handling bug issue in StoreValidator where process hang if remote repository is not valid (#1438) * provide better feedback about bc selection process * typo in stage end brace * record and pass on data tarball to image build in addition to skinny tarball * remove merge error; duplicate else clause * Remove two dup lines * adding trace logging to figure out why raw timings aren't being metered (#1445) * Accumulate timing info for different types of calls in a given user request, then report these in the REST end log (#1446) * skip all tests during deployment (they run earlier) (#1447) * Fix cumulative timing measurement and add accumulation for non-intercepted metrics (#1448) * Add creation time for all repos Seems there is no creation time for all of our current repos. I think it is useful for tracking, especially for the auto-created repos like koji-proxy and implied repos. BTW, I'm also thinking if we also need to add the "last_update_time" for all repos. * Level down metricsManager log for accumulating to trace * Make repo create time as string for ser/de-ser consideration * Switch to calculating cumulative elapsed milliseconds instead of seconds (#1451) * add license headers and bump to release of galley * [maven-release-plugin] prepare release indy-parent-1.9.6 * [maven-release-plugin] prepare for next development iteration * Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager (#1455) * Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager Previously, storing ArtifactStore definitions as JSON requires us to serialize/deserialize (render and parse) constantly, every time we access a repository definition. We can access dozens or even thousands of these in a request, especially in an upload request or a promotion request. Looking at our current production deployment, we see that upload latency is up around 7 seconds at times, with about 6.98 seconds of that being consumed in the process of working with the store-definition cache. This change implements Externalizable for all ArtifactStore derivatives, StoreKey, and related classes in the core model of Indy. It adds some testing to ensure these can be read / written appropriately using readExternal / writeExternal methods, and verifies that round-trip data is preserved. Then, it switches the cache used by InfinispanStoreDataManager to use Cache<StoreKey, ArtifactStore> instead of Cache<StoreKey, String>, and relies on Infinispan itself to use the Externalizable implementations to serialize data when it is persisted to the backing database. Along with this comes a need to migrate existing JSON data to the new binary format. To handle this, I've modified the InfinispanStoreDataMigrationAction class to look for a cache named 'store-data-v1' and if it finds it, to migrate the old-style store definitions from JSON to the new cache, managed by InfinispanStoreDataManager. It will NOT use the really old filesystem storage location if it finds this legacy cache. * make no-arg constructors public and correct usage of Object[In/Out]putStream * Add mixed-lot serialization for ArtifactStore, and read all by ArtifactStore base class This will test that polymorphism works in the object stream. * Cumulative call count in logging MDC and metering ratio (#1457) * remove unused serialization code * implement cumulative counts for meters, and add support for metering ratios to cut down on metering latency * Using store-data-v1 for new object-based store-data cache (#1456) I found a problem for migration process. When we use db-based persistence for store cache, we can not control the table name for which store use by applying a simple "table-name" like attr in ispn xml because it does not support this. The table name is applied by its cache name directly. So we need to use a new cache name for the new obj-based store, ane use the legacy "store-data" cache for migration purpose. * change the current store-data cache name to 'store-data-v2' to imply that 'store-data' is v1 (#1458) * Drill down metrics for store.execute (#1460) * Remove old secrets loading code in indy.sh (#1453) * Rewrite store-query methods that use stream() and intercept/wrap store-query for metrics (#1461) * Rewrite store-query methods that use stream() and intercept/wrap store-query for metrics We cannot use the @measure annotation in ArtifactStoreQuery, since that class is instantiated a variable number of calls for each user request, and instances hold state supporting the fluent API. The StoreDataManager decorator wraps ArtifactStoreQuery results from the query() method in a query class that measures query calls, delegating the actual query to the instance that comes from the underlying StoreDataManager delegate. StoreDataManager stream / getAll methods rely on having in-memory access to ALL of the ArtifactStore instances, and also on having a small enough instance count to make iterating through all of them feasible. In our production environment, both of these assumptions are violated. We need a better way to access this information, but to start I've refactored DefaultArtifactStoreQuery methods that use streams and getAll in the StoreDataManager so they won't do that. In the case of retrieving the stores that are in the membership tree of a particular group, I've rewritten that to avoid recursion. In addition, I've added a keyStream() method to ArtifactStoreQuery and a streamArtifactStoreKeys() method to StoreDataManager. This SHOULD be something that we can optimize, ensuring all keys are available in-memory or easy to deserialize with minimum overhead. We can do a lot of filtering based on the key alone, then additional filtering can be applied with the matching stores, which would be retrieved directly by key from the filtered keys in these calls. * remove duplicate dependencies in 2 poms * Separate metered-db stuff to break dependency cycle, and fix a few compilation / unit test problems * Add license headers * [maven-release-plugin] prepare release indy-parent-1.9.7 * [maven-release-plugin] prepare for next development iteration * Add tck for ispn store data manager And following: * One more tck case for complex group hierarchy for affected by testing * Removed commented code in infinispan data store manager for pretty * FIX: NPE happened during getAffectedBy Seems the new refactored code brings some new way to get affected groups, but it missed some NPE check which caused 500 error. * update license headers in new classes * [maven-release-plugin] prepare release indy-parent-1.9.7.1 * [maven-release-plugin] prepare for next development iteration * Update snapshot version to 1.9.8 * Fix mistakes

@Head

* Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Bump swagger-ui from 2.2.10 to 3.23.11 in /uis/layover Bumps [swagger-ui](https://github.com/swagger-api/swagger-ui) from 2.2.10 to 3.23.11. - [Release notes](https://github.com/swagger-api/swagger-ui/releases) - [Commits](swagger-api/swagger-ui@v2.2.10...v3.23.11) Signed-off-by: dependabot[bot] <support@github.com> * turn off deflate encoding support until we understand why it can corrupt downloaded archives * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques… (#1393) * Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD requests for browsing content - NOS-2048.Also i have rewrite @Head method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception. * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Fix Bad Request 400 error when uploading snapshot maven-metadata (#1389) * Allow disable of content index, and add prometheus metrics adapter Adding Dropwizard Metrics adapter to publish metrics to Prometheus via servlet, since our perf-test instance can't deploy GraphiteDB to a pod. This gets us ready for Prometheus anyway. We may not need content indexing anymore in our particular deployment environment, and it might actually be hurting performance for our use case. Adding an option to disable it via config. * add managed dep on indy-subsys-metrics-prometheus * Fix tests broken by default-disabled content index, and improve some logger names * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Added Tests for checking http HEAD requests on '/api/browse/{packageType}/{type}/{name}/{path (.*)}' * Changes Requested * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Path not directory * Content Digest if path doesn't end with '/' * Added headers in ContentAcessHandler * switch to using openshift api to control/follow image build (#1415) * switch to using openshift api to control/follow image build * fix hook url var to my_bc in Jenkinsfile * provide better feedback about bc selection process * typo in stage end brace * Start path map storage migrator (#1416) * Start path map storage migrator * Fix deps and remove first slash of store path * Bump jacksonVersion from 2.9.10 to 2.10.1 Bumps `jacksonVersion` from 2.9.10 to 2.10.1. Updates `jackson-core` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson-core/releases) - [Commits](FasterXML/jackson-core@jackson-core-2.9.10...jackson-core-2.10.1) Updates `jackson-annotations` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) Updates `jackson-databind` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) Updates `jackson-jaxrs-json-provider` from 2.9.10 to 2.10.1 Signed-off-by: dependabot[bot] <support@github.com> * Add dedupe check support for pathmap migrator * Use move instead of delete for processed path files in todo folder * Add reload processed count for next migrate from failure (#1423) * Fix the path mapped migrator checksum issue (#1424) * Fix TrackedContentEntryTest (#1425) * Avoid one not needed traverse of first level package path And concurrently run scan from package level * Fix groovy.lang.MissingPropertyException: No such property: tarball_url (#1428) * Add filter to filter out the files not need to migrate * [migration-tool] shutdown pathDB after done (#1430) * Remove pathmap-migrator (#1432) Pathmap-migrator has been moved to a single repo here: https://www.github.com/Commonjava/pathmap-migrator * Remove left pathmap-migrator class * Fix the swagger-ui issue when upgraded to 3.23.x (#1435) * Bump keycloakVersion from 6.0.0 to 7.0.1 (#1410) Bumps `keycloakVersion` from 6.0.0 to 7.0.1. Updates `keycloak-undertow-adapter` from 6.0.0 to 7.0.1 Updates `keycloak-core` from 6.0.0 to 7.0.1 - [Release notes](https://github.com/keycloak/keycloak/releases) - [Commits](keycloak/keycloak@6.0.0...7.0.1) Updates `keycloak-adapter-core` from 6.0.0 to 7.0.1 - [Release notes](https://github.com/keycloak/keycloak/releases) - [Commits](keycloak/keycloak@6.0.0...7.0.1) Signed-off-by: dependabot[bot] <support@github.com> * NOS-2167: Handling bug issue in StoreValidator where process hang if remote repository is not valid (#1438) * provide better feedback about bc selection process * typo in stage end brace * record and pass on data tarball to image build in addition to skinny tarball * remove merge error; duplicate else clause * Remove two dup lines * adding trace logging to figure out why raw timings aren't being metered (#1445) * Accumulate timing info for different types of calls in a given user request, then report these in the REST end log (#1446) * skip all tests during deployment (they run earlier) (#1447) * Fix cumulative timing measurement and add accumulation for non-intercepted metrics (#1448) * Add creation time for all repos Seems there is no creation time for all of our current repos. I think it is useful for tracking, especially for the auto-created repos like koji-proxy and implied repos. BTW, I'm also thinking if we also need to add the "last_update_time" for all repos. * Level down metricsManager log for accumulating to trace * Make repo create time as string for ser/de-ser consideration * Switch to calculating cumulative elapsed milliseconds instead of seconds (#1451) * add license headers and bump to release of galley * [maven-release-plugin] prepare release indy-parent-1.9.6 * [maven-release-plugin] prepare for next development iteration * Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager (#1455) * Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager Previously, storing ArtifactStore definitions as JSON requires us to serialize/deserialize (render and parse) constantly, every time we access a repository definition. We can access dozens or even thousands of these in a request, especially in an upload request or a promotion request. Looking at our current production deployment, we see that upload latency is up around 7 seconds at times, with about 6.98 seconds of that being consumed in the process of working with the store-definition cache. This change implements Externalizable for all ArtifactStore derivatives, StoreKey, and related classes in the core model of Indy. It adds some testing to ensure these can be read / written appropriately using readExternal / writeExternal methods, and verifies that round-trip data is preserved. Then, it switches the cache used by InfinispanStoreDataManager to use Cache<StoreKey, ArtifactStore> instead of Cache<StoreKey, String>, and relies on Infinispan itself to use the Externalizable implementations to serialize data when it is persisted to the backing database. Along with this comes a need to migrate existing JSON data to the new binary format. To handle this, I've modified the InfinispanStoreDataMigrationAction class to look for a cache named 'store-data-v1' and if it finds it, to migrate the old-style store definitions from JSON to the new cache, managed by InfinispanStoreDataManager. It will NOT use the really old filesystem storage location if it finds this legacy cache. * make no-arg constructors public and correct usage of Object[In/Out]putStream * Add mixed-lot serialization for ArtifactStore, and read all by ArtifactStore base class This will test that polymorphism works in the object stream. * Cumulative call count in logging MDC and metering ratio (#1457) * remove unused serialization code * implement cumulative counts for meters, and add support for metering ratios to cut down on metering latency * Using store-data-v1 for new object-based store-data cache (#1456) I found a problem for migration process. When we use db-based persistence for store cache, we can not control the table name for which store use by applying a simple "table-name" like attr in ispn xml because it does not support this. The table name is applied by its cache name directly. So we need to use a new cache name for the new obj-based store, ane use the legacy "store-data" cache for migration purpose. * change the current store-data cache name to 'store-data-v2' to imply that 'store-data' is v1 (#1458) * Drill down metrics for store.execute (#1460) * Remove old secrets loading code in indy.sh (#1453) * Rewrite store-query methods that use stream() and intercept/wrap store-query for metrics (#1461) * Rewrite store-query methods that use stream() and intercept/wrap store-query for metrics We cannot use the @measure annotation in ArtifactStoreQuery, since that class is instantiated a variable number of calls for each user request, and instances hold state supporting the fluent API. The StoreDataManager decorator wraps ArtifactStoreQuery results from the query() method in a query class that measures query calls, delegating the actual query to the instance that comes from the underlying StoreDataManager delegate. StoreDataManager stream / getAll methods rely on having in-memory access to ALL of the ArtifactStore instances, and also on having a small enough instance count to make iterating through all of them feasible. In our production environment, both of these assumptions are violated. We need a better way to access this information, but to start I've refactored DefaultArtifactStoreQuery methods that use streams and getAll in the StoreDataManager so they won't do that. In the case of retrieving the stores that are in the membership tree of a particular group, I've rewritten that to avoid recursion. In addition, I've added a keyStream() method to ArtifactStoreQuery and a streamArtifactStoreKeys() method to StoreDataManager. This SHOULD be something that we can optimize, ensuring all keys are available in-memory or easy to deserialize with minimum overhead. We can do a lot of filtering based on the key alone, then additional filtering can be applied with the matching stores, which would be retrieved directly by key from the filtered keys in these calls. * remove duplicate dependencies in 2 poms * Separate metered-db stuff to break dependency cycle, and fix a few compilation / unit test problems * Add license headers * [maven-release-plugin] prepare release indy-parent-1.9.7 * [maven-release-plugin] prepare for next development iteration * Add tck for ispn store data manager And following: * One more tck case for complex group hierarchy for affected by testing * Removed commented code in infinispan data store manager for pretty * FIX: NPE happened during getAffectedBy Seems the new refactored code brings some new way to get affected groups, but it missed some NPE check which caused 500 error. * update license headers in new classes * [maven-release-plugin] prepare release indy-parent-1.9.7.1 * [maven-release-plugin] prepare for next development iteration * Remove unused jdk internal imports * Add comment for new group mangement tck case * Extract getStore operation from deep loop for affected by This getArtifactStore resides in a deep loop, which will cause serious performance problem if the loop size is very huge(Actually it is). So need to extract it out from this loop. * Async the group de-index during upload in content-index service Recent performance problem we met for uploading & promotion is all caused by the getAffectedBy operation, which is a time-consumed one for a full scan on store data. After checking, I found that it is mainly used for de-index or re-index content-index entries for the uploading repo. I think this de-index will not be a block factor during the upload or promotion process, which means we could make it async. In this commit, I created a ISPN cache and used it like a queue, which decouple the de-index opeartion from storing process in content-index to an aync listener of ISPN listener. Why not use a weft executor? Because the storing operation is high frequent, which means it could be blocked again if the de-index effeciency is much slower than the real physical storing operation. Use this queue-liked way we can delegate all these to underlying ISPN and we have a position to set the queue size from ISPN config. * Revert "Async the group de-index during upload in content-index service" This reverts commit 0b3ef58. With load-testing, found that there are a bunch of ispn lock acquiring timeout error, so that means the getAffectedBy operation is much slower than file transferring itself, and cause many unhandled de-index entries in that cache. We need to think about another solution to switch to async mode. * Async the group de-index during upload in content-index service Recent performance problem we met for uploading & promotion is all caused by the getAffectedBy operation, which is a time-consumed one for a full scan on store data. After checking, I found that it is mainly used for de-index or re-index content-index entries for the uploading repo. I think this de-index will not be a block factor during the upload or promotion process, which means we could make it async. In this commit, I switched back to use a normal executor service to async the de-index of the affected group. One potential problem for this solution is the executor queue size, which could cause OOM if there are bunch of uploading happened in a short time period. We need to monitor if this can happen. * Code cleanup: remove dupe and unused deps (#1474) Purpose to reduce indy launcher tarball size. Did following: * Removed all duped junit, harmcrest and cdi delaration * Removed quartz deps as it is not used now * Removed h2 as it is not used now * don't build cache migrator or record extractor in main build. (#1478) These don't need to be on the main release cycle, and it seems there's some recursion problem in the maven-assembly-plugin with one of them that's screwing up Indy builds. * Thread-off two nfc cleaning ops which refers getAffectedBy (#1477) * Thread-off getAffectedBy ops for nfc cleaning in ContentManager * Thread-off getAffectedBy ops for nfc cleaning in promote validation * register NFC cache listener in DefaultContentIndexManager (#1479) * Limit "affected-by" with full scan opeartion in package and type based set (#1471) Add a new Cache which is <package, Map<type, Set<StoreKey>> to store all Store keys based on package and type, so for affected group get method, we can avoid the full scan on all repos to get groups, but just simplified to two map get ops. * Fix NPE when store is null in case of being deleted * Thread-off getAffectedBy ops in NFCContentListener * Extract getGroupsAffectedBy from Query to StoreDataManager * Add reverse map for affected stores by some constituents change * Limit "affected-by" with full scan opeartion in package and type based set (#1471) Add a new Cache which is <package, Map<type, Set<StoreKey>> to store all Store keys based on package and type, so for affected group get method, we can avoid the full scan on all repos to get groups, but just simplified to two map get ops. * Simplify affected-by cache We won't completely flatten out the affected-by cache by resolving groups affected by groups affected by a given store. Instead, we leave it as a simple reverse map of the group's constituents, which will give us direct keys to lookup in order to iterate and calculated these n-level affected relationships. It's still much better than scanning all groups (which in the past meant scanning all repos). At the same time, it will avoid any unforseen problems with group membership changes that don't get propagated via the flattening process. It should be easier to maintain. * populate the affected-by cache on startup if the affected-by cache is clear This should work if the affected-by cache is kept in memory, or for the first run when we use a database to store it. We're lining this up with the by-package-type cache as a single-node fix, where we will still need a more scalable solution from a proper database / data design approach before we're ready for multi-node usage. * clean up affected-by iteration and consolidate iteration in default store query I added a null check and disabled-group check to the iteration for finding groups affected by a particular store. The rest of these are just adjusting away from streams and toward intermediary Set instances...things like that. * fix merge error in StoreDataManager * fix remaining compilation errors left from merging * fix NPE when ssl validation repo is null due to some error * Fix potential NPE by using PackageMetadataMerger as a component, as intended. * Update copyright year range to include 2020, reformat license headers (#1484) * fix license headers * Update project copyright year range to include 2020 and reformat license headers * [maven-release-plugin] prepare release indy-parent-1.9.8 * [maven-release-plugin] prepare for next development iteration * Fix some code smell for PromotionManager like potential NPE Conflicts: api/src/main/java/org/commonjava/indy/data/StoreDataManager.java * Some code simplify Conflicts: db/common/src/main/java/org/commonjava/indy/db/common/AbstractStoreDataManager.java * Code optimization - MavenMetadataGenerator * Simplify lambda * Remove useless null check * Use try resource instead * Charset standardization * Add some trace log for metadata clearing of pom update (#1487) * Fix init of affected-by reverse map, and add checks/error message if it's empty (#1488) * Fix affected-by reversemap initialization in lifecycle startup * Add logging to note when the affected-by cache is altered, and another error log message if it's empty * Refactor InfinispanStoreDataManager.init() to startup actions, add some logging The init() method runs before any migration actions run, which means migrated stores won't be stored in the byPkg cache. The old initializer for the affected-by map has a similar problem, so I removed it in favor of the new startup action. The byPkg init was moved out to a startup action too for this reason. Finally, I've added a log message to show which groups were found to be affected by a specific set of input keys. This will give us better information about whether the affected-by map is being used or not. * adding another ftest that initializes store data at boot instead of at migration * Support honeycomb * Fix config * Add HoneycombInterceptor * Fix getchildspan * Add HoneycombFilter * Add pathInfo to fields * Fix NPE in Honeycomb filter and add some fields to honeycomb spans * Enable collection of more metrics (esp. I/O level) via Honeycomb (#1492) This required refactoring the Galley timing function providers, using a new patch of Galley that attempts to generalize the metrics interface, at least for I/O operations. It also involved creating three new interceptors, which would handle methods that are designed to wrap arbitrary pieces of code in metrics. * return the Honeycomb trace span at the filter level, not the root span that is inside it. (#1493) * add golden signals filter into IndyDeployment because filter ordering is important (#1494) * fix cyclic dependency caused by moving GoldenSignalsFilter (#1495) * fix cyclic dependency caused by moving GoldenSignalsFilter * Adding beans.xml to allow CDI injection properly * upgrade galley to release and fix license headers * [maven-release-plugin] prepare release indy-parent-1.9.9 * [maven-release-plugin] prepare for next development iteration * Honeycomb field-capture improvements, and span configuration (#1497) * Fix the Honeycomb root span (which I broke previously) * debug some honeycomb field-tracking problems I had to break out the ThreadContext initialization into an outer servlet filter (specified first in the mapping order, since it's the outer one) to preserve fields until honeycomb can find it. I've also added the ability to turn spans on/off in the configuration, using EITHER include OR exclude semantics (you can't use both). * ignore metrics testing for now...test needs to be updated * Consolidate all async getGroupAffectedBy ops in StoreDataManager * New Relic telemetry support (#1498) * Add New Relic telemetry support, and complete configuration of interceptors in beans.xml for Honeycomb * check before sending batch, push field extraction to manager, use Dequeue instead of List * Add New Relic telemetry support, and complete configuration of interceptors in beans.xml for Honeycomb * check before sending batch, push field extraction to manager, use Dequeue instead of List * Refine / fix new relic span tracking * try to forget we ever used New Relic tracing, the UI is terrible * Add new configuration approach for honeycomb and configurable sampling rates Sampling rates can be specified by Golden Signals (SLI) function name, or by method name, or by class name + method name, etc. * forgot to add configuration parameter for base sample rate * fix compilation error * Tie in threaded-off calls to Honeycomb trace by simulating jump to another service * refactored thread contextualizer for honeycomb I'm ignoring the client bound to other threads and just creating a new one from config when needed. This should work around NPEs in deeper / uncontrolled threads. * Align version to 1.9.10-SNAPSHOT * Update license head * Fix merge issues * Fix merge issues-2 * Fix 2 ftests where the bytestream is drained after the first use * Revert "Fix 2 ftests where the bytestream is drained after the first use" This reverts commit 81df08a. Co-authored-by: Wenjie Guo <guowj12@gmail.com> Co-authored-by: georgy <3187346+gorgija@users.noreply.github.com> Co-authored-by: Gang Li <ligangty@gmail.com> Co-authored-by: John Casey <jdcasey@commonjava.org> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: geored <47973913+geored@users.noreply.github.com>

* Integrate with pathmapped storage (#1396) * Integrate with pathmapped storage * Start embedded cassandra in AbstractIndyFunctionalTest * Add missing configs * Start cassandra before fixture server (#1398) * Fix ftest about how to check storage file existence (#1400) * Fix ftest cassandra keyspace and use random port (#1401) * Fix ftest cassandra keyspace and use random port * Fix another two ftests for how to check physical file * Fix getStorageFile by find the latest under hashed dir * Fix some ftests where the initBaseConfig were overriden * Apply cassandra-maven-plugin for ftests (#1402) * Apply cassandra-maven-plugin for ftests * Fix another four ftests via getPhysicalPath * Fix the remaining ftests which were failed due to the change to file check (#1404) * Add subsys/cassandra for general connection settings (#1411) * Add subsys/cassandra for general connection settings * Fix GroupMetadataFileDeletionTest the way it checks file exists * Remove test keyspace to release resources * Handle connect error * Ignore drop keyspace errors * Create pathDB with Cassandra client session * Drop keyspace before session closed * Master integration pathmap mergemaster (#1412) * Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Bump swagger-ui from 2.2.10 to 3.23.11 in /uis/layover Bumps [swagger-ui](https://github.com/swagger-api/swagger-ui) from 2.2.10 to 3.23.11. - [Release notes](https://github.com/swagger-api/swagger-ui/releases) - [Commits](https://github.com/swagger-api/swagger-ui/compare/v2.2.10...v3.23.11) Signed-off-by: dependabot[bot] <support@github.com> * turn off deflate encoding support until we understand why it can corrupt downloaded archives * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques… (#1393) * Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD requests for browsing content - NOS-2048.Also i have rewrite @HEAD method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception. * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Fix Bad Request 400 error when uploading snapshot maven-metadata (#1389) * Allow disable of content index, and add prometheus metrics adapter Adding Dropwizard Metrics adapter to publish metrics to Prometheus via servlet, since our perf-test instance can't deploy GraphiteDB to a pod. This gets us ready for Prometheus anyway. We may not need content indexing anymore in our particular deployment environment, and it might actually be hurting performance for our use case. Adding an option to disable it via config. * add managed dep on indy-subsys-metrics-prometheus * Fix tests broken by default-disabled content index, and improve some logger names * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Added Tests for checking http HEAD requests on '/api/browse/{packageType}/{type}/{name}/{path (.*)}' * Changes Requested * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Path not directory * Content Digest if path doesn't end with '/' * Added headers in ContentAcessHandler * Master integration pathmap rebase (#1413) * Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Bump swagger-ui from 2.2.10 to 3.23.11 in /uis/layover Bumps [swagger-ui](https://github.com/swagger-api/swagger-ui) from 2.2.10 to 3.23.11. - [Release notes](https://github.com/swagger-api/swagger-ui/releases) - [Commits](https://github.com/swagger-api/swagger-ui/compare/v2.2.10...v3.23.11) Signed-off-by: dependabot[bot] <support@github.com> * turn off deflate encoding support until we understand why it can corrupt downloaded archives * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques… (#1393) * Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD requests for browsing content - NOS-2048.Also i have rewrite @HEAD method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception. * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Fix Bad Request 400 error when uploading snapshot maven-metadata (#1389) * Allow disable of content index, and add prometheus metrics adapter Adding Dropwizard Metrics adapter to publish metrics to Prometheus via servlet, since our perf-test instance can't deploy GraphiteDB to a pod. This gets us ready for Prometheus anyway. We may not need content indexing anymore in our particular deployment environment, and it might actually be hurting performance for our use case. Adding an option to disable it via config. * add managed dep on indy-subsys-metrics-prometheus * Fix tests broken by default-disabled content index, and improve some logger names * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Added Tests for checking http HEAD requests on '/api/browse/{packageType}/{type}/{name}/{path (.*)}' * Changes Requested * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Path not directory * Content Digest if path doesn't end with '/' * Added headers in ContentAcessHandler * Integrate with pathmapped storage (#1396) * Integrate with pathmapped storage * Start embedded cassandra in AbstractIndyFunctionalTest * Add missing configs * Start cassandra before fixture server (#1398) * Fix ftest about how to check storage file existence (#1400) * Fix ftest cassandra keyspace and use random port (#1401) * Fix ftest cassandra keyspace and use random port * Fix another two ftests for how to check physical file * Fix getStorageFile by find the latest under hashed dir * Fix some ftests where the initBaseConfig were overriden * Apply cassandra-maven-plugin for ftests (#1402) * Apply cassandra-maven-plugin for ftests * Fix another four ftests via getPhysicalPath * Fix the remaining ftests which were failed due to the change to file check (#1404) * Add subsys/cassandra for general connection settings (#1411) * Add subsys/cassandra for general connection settings * Fix GroupMetadataFileDeletionTest the way it checks file exists * Remove test keyspace to release resources * Handle connect error * Ignore drop keyspace errors * Create pathDB with Cassandra client session * Drop keyspace before session closed * Measure pathDB operations (#1414) * Adapt to path-mapped package changes (#1417) * Add confset for subsys/cassandra (#1419) * Adapt to path-mapped package changes * Add confset for subsys/cassandra * switch to using openshift api to control/follow image build (#1415) (#1418) * switch to using openshift api to control/follow image build * fix hook url var to my_bc in Jenkinsfile * provide better feedback about bc selection process * typo in stage end brace * Set cassandraClient so-connection timeout and add close method (#1422) * Add CassandraClient.close * Add a debug log * Extend cassandra cluster maxMem * Back to standalone cassandra * Put off close * Try again by only close session * use cassandraClient.close * Fix TrackedContentEntryTest * setConnectTimeoutMillis to 30 seconds * Fix groovy.lang.MissingPropertyException: No such property: tarball_url (#1427) * Fix npe in CassandraClient (#1431) * Fix npe in CassandraClient * Set socket options * Addon path-mapped to provide direct listing/get/delete operations (#1436) * Cherry fixes for Swagger and bump some versions (#1437) * Fix the swagger-ui issue when upgraded to 3.23.x (#1435) * Bump keycloakVersion from 6.0.0 to 7.0.1 (#1410) Bumps `keycloakVersion` from 6.0.0 to 7.0.1. Updates `keycloak-undertow-adapter` from 6.0.0 to 7.0.1 Updates `keycloak-core` from 6.0.0 to 7.0.1 - [Release notes](https://github.com/keycloak/keycloak/releases) - [Commits](https://github.com/keycloak/keycloak/compare/6.0.0...7.0.1) Updates `keycloak-adapter-core` from 6.0.0 to 7.0.1 - [Release notes](https://github.com/keycloak/keycloak/releases) - [Commits](https://github.com/keycloak/keycloak/compare/6.0.0...7.0.1) Signed-off-by: dependabot[bot] <support@github.com> * Add query param limit to pathmapresource.list (#1439) * Filter listing result by query param type (file,dir,all) (#1441) * Filter listing result by query param type (file,dir,all) * Update the filter * Remove useless import * Update * Fix null fileType * Merge master to path-mapped feature branch (#1459) * Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Bump swagger-ui from 2.2.10 to 3.23.11 in /uis/layover Bumps [swagger-ui](https://github.com/swagger-api/swagger-ui) from 2.2.10 to 3.23.11. - [Release notes](https://github.com/swagger-api/swagger-ui/releases) - [Commits](https://github.com/swagger-api/swagger-ui/compare/v2.2.10...v3.23.11) Signed-off-by: dependabot[bot] <support@github.com> * turn off deflate encoding support until we understand why it can corrupt downloaded archives * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques… (#1393) * Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD requests for browsing content - NOS-2048.Also i have rewrite @HEAD method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception. * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Fix Bad Request 400 error when uploading snapshot maven-metadata (#1389) * Allow disable of content index, and add prometheus metrics adapter Adding Dropwizard Metrics adapter to publish metrics to Prometheus via servlet, since our perf-test instance can't deploy GraphiteDB to a pod. This gets us ready for Prometheus anyway. We may not need content indexing anymore in our particular deployment environment, and it might actually be hurting performance for our use case. Adding an option to disable it via config. * add managed dep on indy-subsys-metrics-prometheus * Fix tests broken by default-disabled content index, and improve some logger names * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Added Tests for checking http HEAD requests on '/api/browse/{packageType}/{type}/{name}/{path (.*)}' * Changes Requested * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Path not directory * Content Digest if path doesn't end with '/' * Added headers in ContentAcessHandler * switch to using openshift api to control/follow image build (#1415) * switch to using openshift api to control/follow image build * fix hook url var to my_bc in Jenkinsfile * provide better feedback about bc selection process * typo in stage end brace * Start path map storage migrator (#1416) * Start path map storage migrator * Fix deps and remove first slash of store path * Bump jacksonVersion from 2.9.10 to 2.10.1 Bumps `jacksonVersion` from 2.9.10 to 2.10.1. Updates `jackson-core` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson-core/releases) - [Commits](https://github.com/FasterXML/jackson-core/compare/jackson-core-2.9.10...jackson-core-2.10.1) Updates `jackson-annotations` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) Updates `jackson-databind` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) Updates `jackson-jaxrs-json-provider` from 2.9.10 to 2.10.1 Signed-off-by: dependabot[bot] <support@github.com> * Add dedupe check support for pathmap migrator * Use move instead of delete for processed path files in todo folder * Add reload processed count for next migrate from failure (#1423) * Fix the path mapped migrator checksum issue (#1424) * Fix TrackedContentEntryTest (#1425) * Avoid one not needed traverse of first level package path And concurrently run scan from package level * Fix groovy.lang.MissingPropertyException: No such property: tarball_url (#1428) * Add filter to filter out the files not need to migrate * [migration-tool] shutdown pathDB after done (#1430) * Remove pathmap-migrator (#1432) Pathmap-migrator has been moved to a single repo here: https://www.github.com/Commonjava/pathmap-migrator * Remove left pathmap-migrator class * Fix the swagger-ui issue when upgraded to 3.23.x (#1435) * Bump keycloakVersion from 6.0.0 to 7.0.1 (#1410) Bumps `keycloakVersion` from 6.0.0 to 7.0.1. Updates `keycloak-undertow-adapter` from 6.0.0 to 7.0.1 Updates `keycloak-core` from 6.0.0 to 7.0.1 - [Release notes](https://github.com/keycloak/keycloak/releases) - [Commits](https://github.com/keycloak/keycloak/compare/6.0.0...7.0.1) Updates `keycloak-adapter-core` from 6.0.0 to 7.0.1 - [Release notes](https://github.com/keycloak/keycloak/releases) - [Commits](https://github.com/keycloak/keycloak/compare/6.0.0...7.0.1) Signed-off-by: dependabot[bot] <support@github.com> * NOS-2167: Handling bug issue in StoreValidator where process hang if remote repository is not valid (#1438) * provide better feedback about bc selection process * typo in stage end brace * record and pass on data tarball to image build in addition to skinny tarball * remove merge error; duplicate else clause * Remove two dup lines * adding trace logging to figure out why raw timings aren't being metered (#1445) * Accumulate timing info for different types of calls in a given user request, then report these in the REST end log (#1446) * skip all tests during deployment (they run earlier) (#1447) * Fix cumulative timing measurement and add accumulation for non-intercepted metrics (#1448) * Add creation time for all repos Seems there is no creation time for all of our current repos. I think it is useful for tracking, especially for the auto-created repos like koji-proxy and implied repos. BTW, I'm also thinking if we also need to add the "last_update_time" for all repos. * Level down metricsManager log for accumulating to trace * Make repo create time as string for ser/de-ser consideration * Switch to calculating cumulative elapsed milliseconds instead of seconds (#1451) * add license headers and bump to release of galley * [maven-release-plugin] prepare release indy-parent-1.9.6 * [maven-release-plugin] prepare for next development iteration * Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager (#1455) * Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager Previously, storing ArtifactStore definitions as JSON requires us to serialize/deserialize (render and parse) constantly, every time we access a repository definition. We can access dozens or even thousands of these in a request, especially in an upload request or a promotion request. Looking at our current production deployment, we see that upload latency is up around 7 seconds at times, with about 6.98 seconds of that being consumed in the process of working with the store-definition cache. This change implements Externalizable for all ArtifactStore derivatives, StoreKey, and related classes in the core model of Indy. It adds some testing to ensure these can be read / written appropriately using readExternal / writeExternal methods, and verifies that round-trip data is preserved. Then, it switches the cache used by InfinispanStoreDataManager to use Cache<StoreKey, ArtifactStore> instead of Cache<StoreKey, String>, and relies on Infinispan itself to use the Externalizable implementations to serialize data when it is persisted to the backing database. Along with this comes a need to migrate existing JSON data to the new binary format. To handle this, I've modified the InfinispanStoreDataMigrationAction class to look for a cache named 'store-data-v1' and if it finds it, to migrate the old-style store definitions from JSON to the new cache, managed by InfinispanStoreDataManager. It will NOT use the really old filesystem storage location if it finds this legacy cache. * make no-arg constructors public and correct usage of Object[In/Out]putStream * Add mixed-lot serialization for ArtifactStore, and read all by ArtifactStore base class This will test that polymorphism works in the object stream. * Cumulative call count in logging MDC and metering ratio (#1457) * remove unused serialization code * implement cumulative counts for meters, and add support for metering ratios to cut down on metering latency * Using store-data-v1 for new object-based store-data cache (#1456) I found a problem for migration process. When we use db-based persistence for store cache, we can not control the table name for which store use by applying a simple "table-name" like attr in ispn xml because it does not support this. The table name is applied by its cache name directly. So we need to use a new cache name for the new obj-based store, ane use the legacy "store-data" cache for migration purpose. * change the current store-data cache name to 'store-data-v2' to imply that 'store-data' is v1 (#1458) * Merge master to pathmap (1.9.8-SNAPSHOT) (#1464) * Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Bump swagger-ui from 2.2.10 to 3.23.11 in /uis/layover Bumps [swagger-ui](https://github.com/swagger-api/swagger-ui) from 2.2.10 to 3.23.11. - [Release notes](https://github.com/swagger-api/swagger-ui/releases) - [Commits](https://github.com/swagger-api/swagger-ui/compare/v2.2.10...v3.23.11) Signed-off-by: dependabot[bot] <support@github.com> * turn off deflate encoding support until we understand why it can corrupt downloaded archives * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques… (#1393) * Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD requests for browsing content - NOS-2048.Also i have rewrite @HEAD method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception. * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Fix Bad Request 400 error when uploading snapshot maven-metadata (#1389) * Allow disable of content index, and add prometheus metrics adapter Adding Dropwizard Metrics adapter to publish metrics to Prometheus via servlet, since our perf-test instance can't deploy GraphiteDB to a pod. This gets us ready for Prometheus anyway. We may not need content indexing anymore in our particular deployment environment, and it might actually be hurting performance for our use case. Adding an option to disable it via config. * add managed dep on indy-subsys-metrics-prometheus * Fix tests broken by default-disabled content index, and improve some logger names * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Added Tests for checking http HEAD requests on '/api/browse/{packageType}/{type}/{name}/{path (.*)}' * Changes Requested * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Path not directory * Content Digest if path doesn't end with '/' * Added headers in ContentAcessHandler * switch to using openshift api to control/follow image build (#1415) * switch to using openshift api to control/follow image build * fix hook url var to my_bc in Jenkinsfile * provide better feedback about bc selection process * typo in stage end brace * Start path map storage migrator (#1416) * Start path map storage migrator * Fix deps and remove first slash of store path * Bump jacksonVersion from 2.9.10 to 2.10.1 Bumps `jacksonVersion` from 2.9.10 to 2.10.1. Updates `jackson-core` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson-core/releases) - [Commits](https://github.com/FasterXML/jackson-core/compare/jackson-core-2.9.10...jackson-core-2.10.1) Updates `jackson-annotations` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) Updates `jackson-databind` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) Updates `jackson-jaxrs-json-provider` from 2.9.10 to 2.10.1 Signed-off-by: dependabot[bot] <support@github.com> * Add dedupe check support for pathmap migrator * Use move instead of delete for processed path files in todo folder * Add reload processed count for next migrate from failure (#1423) * Fix the path mapped migrator checksum issue (#1424) * Fix TrackedContentEntryTest (#1425) * Avoid one not needed traverse of first level package path And concurrently run scan from package level * Fix groovy.lang.MissingPropertyException: No such property: tarball_url (#1428) * Add filter to filter out the files not need to migrate * [migration-tool] shutdown pathDB after done (#1430) * Remove pathmap-migrator (#1432) Pathmap-migrator has been moved to a single repo here: https://www.github.com/Commonjava/pathmap-migrator * Remove left pathmap-migrator class * Fix the swagger-ui issue when upgraded to 3.23.x (#1435) * Bump keycloakVersion from 6.0.0 to 7.0.1 (#1410) Bumps `keycloakVersion` from 6.0.0 to 7.0.1. Updates `keycloak-undertow-adapter` from 6.0.0 to 7.0.1 Updates `keycloak-core` from 6.0.0 to 7.0.1 - [Release notes](https://github.com/keycloak/keycloak/releases) - [Commits](https://github.com/keycloak/keycloak/compare/6.0.0...7.0.1) Updates `keycloak-adapter-core` from 6.0.0 to 7.0.1 - [Release notes](https://github.com/keycloak/keycloak/releases) - [Commits](https://github.com/keycloak/keycloak/compare/6.0.0...7.0.1) Signed-off-by: dependabot[bot] <support@github.com> * NOS-2167: Handling bug issue in StoreValidator where process hang if remote repository is not valid (#1438) * provide better feedback about bc selection process * typo in stage end brace * record and pass on data tarball to image build in addition to skinny tarball * remove merge error; duplicate else clause * Remove two dup lines * adding trace logging to figure out why raw timings aren't being metered (#1445) * Accumulate timing info for different types of calls in a given user request, then report these in the REST end log (#1446) * skip all tests during deployment (they run earlier) (#1447) * Fix cumulative timing measurement and add accumulation for non-intercepted metrics (#1448) * Add creation time for all repos Seems there is no creation time for all of our current repos. I think it is useful for tracking, especially for the auto-created repos like koji-proxy and implied repos. BTW, I'm also thinking if we also need to add the "last_update_time" for all repos. * Level down metricsManager log for accumulating to trace * Make repo create time as string for ser/de-ser consideration * Switch to calculating cumulative elapsed milliseconds instead of seconds (#1451) * add license headers and bump to release of galley * [maven-release-plugin] prepare release indy-parent-1.9.6 * [maven-release-plugin] prepare for next development iteration * Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager (#1455) * Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager Previously, storing ArtifactStore definitions as JSON requires us to serialize/deserialize (render and parse) constantly, every time we access a repository definition. We can access dozens or even thousands of these in a request, especially in an upload request or a promotion request. Looking at our current production deployment, we see that upload latency is up around 7 seconds at times, with about 6.98 seconds of that being consumed in the process of working with the store-definition cache. This change implements Externalizable for all ArtifactStore derivatives, StoreKey, and related classes in the core model of Indy. It adds some testing to ensure these can be read / written appropriately using readExternal / writeExternal methods, and verifies that round-trip data is preserved. Then, it switches the cache used by InfinispanStoreDataManager to use Cache<StoreKey, ArtifactStore> instead of Cache<StoreKey, String>, and relies on Infinispan itself to use the Externalizable implementations to serialize data when it is persisted to the backing database. Along with this comes a need to migrate existing JSON data to the new binary format. To handle this, I've modified the InfinispanStoreDataMigrationAction class to look for a cache named 'store-data-v1' and if it finds it, to migrate the old-style store definitions from JSON to the new cache, managed by InfinispanStoreDataManager. It will NOT use the really old filesystem storage location if it finds this legacy cache. * make no-arg constructors public and correct usage of Object[In/Out]putStream * Add mixed-lot serialization for ArtifactStore, and read all by ArtifactStore base class This will test that polymorphism works in the object stream. * Cumulative call count in logging MDC and metering ratio (#1457) * remove unused serialization code * implement cumulative counts for meters, and add support for metering ratios to cut down on metering latency * Using store-data-v1 for new object-based store-data cache (#1456) I found a problem for migration process. When we use db-based persistence for store cache, we can not control the table name for which store use by applying a simple "table-name" like attr in ispn xml because it does not support this. The table name is applied by its cache name directly. So we need to use a new cache name for the new obj-based store, ane use the legacy "store-data" cache for migration purpose. * change the current store-data cache name to 'store-data-v2' to imply that 'store-data' is v1 (#1458) * Drill down metrics for store.execute (#1460) * Remove old secrets loading code in indy.sh (#1453) * Rewrite store-query methods that use stream() and intercept/wrap store-query for metrics (#1461) * Rewrite store-query methods that use stream() and intercept/wrap store-query for metrics We cannot use the @Measure annotation in ArtifactStoreQuery, since that class is instantiated a variable number of calls for each user request, and instances hold state supporting the fluent API. The StoreDataManager decorator wraps ArtifactStoreQuery results from the query() method in a query class that measures query calls, delegating the actual query to the instance that comes from the underlying StoreDataManager delegate. StoreDataManager stream / getAll methods rely on having in-memory access to ALL of the ArtifactStore instances, and also on having a small enough instance count to make iterating through all of them feasible. In our production environment, both of these assumptions are violated. We need a better way to access this information, but to start I've refactored DefaultArtifactStoreQuery methods that use streams and getAll in the StoreDataManager so they won't do that. In the case of retrieving the stores that are in the membership tree of a particular group, I've rewritten that to avoid recursion. In addition, I've added a keyStream() method to ArtifactStoreQuery and a streamArtifactStoreKeys() method to StoreDataManager. This SHOULD be something that we can optimize, ensuring all keys are available in-memory or easy to deserialize with minimum overhead. We can do a lot of filtering based on the key alone, then additional filtering can be applied with the matching stores, which would be retrieved directly by key from the filtered keys in these calls. * remove duplicate dependencies in 2 poms * Separate metered-db stuff to break dependency cycle, and fix a few compilation / unit test problems * Add license headers * [maven-release-plugin] prepare release indy-parent-1.9.7 * [maven-release-plugin] prepare for next development iteration * Add tck for ispn store data manager And following: * One more tck case for complex group hierarchy for affected by testing * Removed commented code in infinispan data store manager for pretty * FIX: NPE happened during getAffectedBy Seems the new refactored code brings some new way to get affected groups, but it missed some NPE check which caused 500 error. * update license headers in new classes * [maven-release-plugin] prepare release indy-parent-1.9.7.1 * [maven-release-plugin] prepare for next development iteration * Update snapshot version to 1.9.8 * Fix mistakes * Add path-mapped-storage dep version to indy pom (#1466) * Make cassandra-maven-plugin run only in ci mode * Merge master to integration pathmap (#1467) * Merge master to pathmap branch (#1501) * Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Bump swagger-ui from 2.2.10 to 3.23.11 in /uis/layover Bumps [swagger-ui](https://github.com/swagger-api/swagger-ui) from 2.2.10 to 3.23.11. - [Release notes](https://github.com/swagger-api/swagger-ui/releases) - [Commits](https://github.com/swagger-api/swagger-ui/compare/v2.2.10...v3.23.11) Signed-off-by: dependabot[bot] <support@github.com> * turn off deflate encoding support until we understand why it can corrupt downloaded archives * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD reques… (#1393) * Reuse the existing repo as implied one should add it into group * NOS-2099: Added information from validating RemoteRepositories inside RemoteRepository metadata object. Also removing disabling RemoteRepository because that will be job done from Adhok-Repo Migration Service. * Test * Added MD5 and SHA1 digested hashes inside header for HTTP HEAD requests for browsing content - NOS-2048.Also i have rewrite @HEAD method for '/api/browse/{packageType}/{type}/{name}/{path}' endpoint because for some reason previously was throwing 500 internal exception. * Filter the existing repos matching the conditions * Enhance the condition for comparing the repo * Fix Bad Request 400 error when uploading snapshot maven-metadata (#1389) * Allow disable of content index, and add prometheus metrics adapter Adding Dropwizard Metrics adapter to publish metrics to Prometheus via servlet, since our perf-test instance can't deploy GraphiteDB to a pod. This gets us ready for Prometheus anyway. We may not need content indexing anymore in our particular deployment environment, and it might actually be hurting performance for our use case. Adding an option to disable it via config. * add managed dep on indy-subsys-metrics-prometheus * Fix tests broken by default-disabled content index, and improve some logger names * Deprecated listContents in client module as it is not working now * Improve Prometheus metrics using node_prefix as a label to help with variables for Grafana * Tweak the condition and merge the keys of metadata "implied_by_stores" * Added Tests for checking http HEAD requests on '/api/browse/{packageType}/{type}/{name}/{path (.*)}' * Changes Requested * Changes Required * only do deploy when we're doing an image build, and add timestamps to Folo tracking records * Adding unit tests for version change in TrackedContentEntry * Bug fix * InternalFeature isn't null value * DefaultStoreValidatorTest * Fixing merge logic in TrackedContentEntry and fixing serialVersionUID so we can use Externalizable versioning for deser * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Added Digesting Algos for Repository contents inside 'ResponseHelper' class * Path not directory * Content Digest if path doesn't end with '/' * Added headers in ContentAcessHandler * switch to using openshift api to control/follow image build (#1415) * switch to using openshift api to control/follow image build * fix hook url var to my_bc in Jenkinsfile * provide better feedback about bc selection process * typo in stage end brace * Start path map storage migrator (#1416) * Start path map storage migrator * Fix deps and remove first slash of store path * Bump jacksonVersion from 2.9.10 to 2.10.1 Bumps `jacksonVersion` from 2.9.10 to 2.10.1. Updates `jackson-core` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson-core/releases) - [Commits](https://github.com/FasterXML/jackson-core/compare/jackson-core-2.9.10...jackson-core-2.10.1) Updates `jackson-annotations` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) Updates `jackson-databind` from 2.9.10 to 2.10.1 - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) Updates `jackson-jaxrs-json-provider` from 2.9.10 to 2.10.1 Signed-off-by: dependabot[bot] <support@github.com> * Add dedupe check support for pathmap migrator * Use move instead of delete for processed path files in todo folder * Add reload processed count for next migrate from failure (#1423) * Fix the path mapped migrator checksum issue (#1424) * Fix TrackedContentEntryTest (#1425) * Avoid one not needed traverse of first level package path And concurrently run scan from package level * Fix groovy.lang.MissingPropertyException: No such property: tarball_url (#1428) * Add filter to filter out the files not need to migrate * [migration-tool] shutdown pathDB after done (#1430) * Remove pathmap-migrator (#1432) Pathmap-migrator has been moved to a single repo here: https://www.github.com/Commonjava/pathmap-migrator * Remove left pathmap-migrator class * Fix the swagger-ui issue when upgraded to 3.23.x (#1435) * Bump keycloakVersion from 6.0.0 to 7.0.1 (#1410) Bumps `keycloakVersion` from 6.0.0 to 7.0.1. Updates `keycloak-undertow-adapter` from 6.0.0 to 7.0.1 Updates `keycloak-core` from 6.0.0 to 7.0.1 - [Release notes](https://github.com/keycloak/keycloak/releases) - [Commits](https://github.com/keycloak/keycloak/compare/6.0.0...7.0.1) Updates `keycloak-adapter-core` from 6.0.0 to 7.0.1 - [Release notes](https://github.com/keycloak/keycloak/releases) - [Commits](https://github.com/keycloak/keycloak/compare/6.0.0...7.0.1) Signed-off-by: dependabot[bot] <support@github.com> * NOS-2167: Handling bug issue in StoreValidator where process hang if remote repository is not valid (#1438) * provide better feedback about bc selection process * typo in stage end brace * record and pass on data tarball to image build in addition to skinny tarball * remove merge error; duplicate else clause * Remove two dup lines * adding trace logging to figure out why raw timings aren't being metered (#1445) * Accumulate timing info for different types of calls in a given user request, then report these in the REST end log (#1446) * skip all tests during deployment (they run earlier) (#1447) * Fix cumulative timing measurement and add accumulation for non-intercepted metrics (#1448) * Add creation time for all repos Seems there is no creation time for all of our current repos. I think it is useful for tracking, especially for the auto-created repos like koji-proxy and implied repos. BTW, I'm also thinking if we also need to add the "last_update_time" for all repos. * Level down metricsManager log for accumulating to trace * Make repo create time as string for ser/de-ser consideration * Switch to calculating cumulative elapsed milliseconds instead of seconds (#1451) * add license headers and bump to release of galley * [maven-release-plugin] prepare release indy-parent-1.9.6 * [maven-release-plugin] prepare for next development iteration * Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager (#1455) * Switch to using Externalizable/ArtifactStore for InfinispanStoreDataManager Previously, storing ArtifactStore definitions as JSON requires us to serialize/deserialize (render and parse) constantly, every time we access a repository definition. We can access dozens or even thousands of these in a request, especially in an upload request or a promotion request. Looking at our current production deployment, we see that upload latency is up around 7 seconds at times, with about 6.98 seconds of that being consumed in the process of working with the store-definition cache. This change implements Externalizable for all ArtifactStore derivatives, StoreKey, and related classes in the core model of Indy. It adds some testing to ensure these can be read / written appropriately using readExternal / writeExternal methods, and verifies that round-trip data is preserved. Then, it switches the cache used by InfinispanStoreDataManager to use Cache<StoreKey, ArtifactStore> instead of Cache<StoreKey, String>, and relies on Infinispan itself to use the Externalizable implementations to serialize data when it is persisted to the backing database. Along with this comes a need to migrate existing JSON data to the new binary format. To handle this, I've modified the InfinispanStoreDataMigrationAction class to look for a cache named 'store-data-v1' and if it finds it, to migrate the old-style store definitions from JSON to the new cache, managed by InfinispanStoreDataManager. It will NOT use the really old filesystem storage location if it finds this legacy cache. * make no-arg constructors public and correct usage of Object[In/Out]putStream * Add mixed-lot serialization for ArtifactStore, and read all by ArtifactStore base class This will test that polymorphism works in the object stream. * Cumulative call count in logging MDC and metering ratio (#1457) * remove unused serialization code * implement cumulative counts for meters, and add support for metering ratios to cut down on metering latency * Using store-data-v1 for new object-based store-data cache (#1456) I found a problem for migration process. When we use db-based persistence for store cache, we can not control the table name for which store use by applying a simple "table-name" like attr in ispn xml because it does not support this. The table name is applied by its cache name directly. So we need to use a new cache name for the new obj-based store, ane use the legacy "store-data" cache for migration purpose. * change the current store-data cache name to 'store-data-v2' to imply that 'store-data' is v1 (#1458) * Drill down metrics for store.execute (#1460) * Remove old secrets loading code in indy.sh (#1453) * Rewrite store-query methods that use stream() and intercept/wrap store-query for metrics (#1461) * Rewrite store-query methods that use stream() and intercept/wrap store-query for metrics We cannot use the @Measure annotation in ArtifactStoreQuery, since that class is instantiated a variable number of calls for each user request, and instances hold state supporting the fluent API. The StoreDataManager decorator wraps ArtifactStoreQuery results from the query() method in a query class that measures query calls, delegating the actual query to the instance that comes from the underlying StoreDataManager delegate. StoreDataManager stream / getAll methods rely on having in-memory access to ALL of the ArtifactStore instances, and also on having a small enough instance count to make iterating through all of them feasible. In our production environment, both of these assumptions are violated. We need a better way to access this information, but to start I've refactored DefaultArtifactStoreQuery methods that use streams and getAll in the StoreDataManager so they won't do that. In the case of retrieving the stores that are in the membership tree of a particular group, I've rewritten that to avoid recursion. In addition, I've added a keyStream() method to ArtifactStoreQuery and a streamArtifactStoreKeys() method to StoreDataManager. This SHOULD be something that we can optimize, ensuring all keys are available in-memory or easy to deserialize with minimum overhead. We can do a lot of filtering based on the key alone, then additional filtering can be applied with the matching stores, which would be retrieved directly by key from the filtered keys in these calls. * remove duplicate dependencies in 2 poms * Separate metered-db stuff to break dependency cycle, and fix a few compilation / unit test problems * Add license headers * [maven-release-plugin] prepare release indy-parent-1.9.7 * [maven-release-plugin] prepare for next development iteration * Add tck for ispn store data manager And following: * One more tck case for complex group hierarchy for affected by testing * Removed commented code in infinispan data store manager for pretty * FIX: NPE happened during getAffectedBy Seems the new refactored code brings some new way to get affected groups, but it missed some NPE check which caused 500 error. * update license headers in new classes * [maven-release-plugin] prepare release indy-parent-1.9.7.1 * [maven-release-plugin] prepare for next development iteration * Remove unused jdk internal imports * Add comment for new group mangement tck case * Extract getStore operation from deep loop for affected by This getArtifactStore resides in a deep loop, which will cause serious performance problem if the loop size is very huge(Actually it is). So need to extract it out from this loop. * Async the group de-index during upload in content-index service Recent performance problem we met for uploading & promotion is all caused by the getAffectedBy operation, which is a time-consumed one for a full scan on store data. After checking, I found that it is mainly used for de-index or re-index content-index entries for the uploading repo. I think this de-index will not be a block factor during the upload or promotion process, which means we could make it async. In this commit, I created a ISPN cache and used it like a queue, which decouple the de-index opeartion from storing process in content-index to an aync listener of ISPN listener. Why not use a weft executor? Because the storing operation is high frequent, which means it could be blocked again if the de-index effeciency is much slower than the real physical storing operation. Use this queue-liked way we can delegate all these to underlying ISPN and we have a position to set the queue size from ISPN config. * Revert "Async the group de-index during upload in content-index service" This reverts commit 0b3ef58f054dab2f336d172ba0e839429b339b5e. With load-testing, found that there are a bunch of ispn lock acquiring timeout error, so that means the getAffectedBy operation is much slower than file transferring itself, and cause many unhandled de-index entries in that cache. We need to think about another solution to switch to async mode. * Async the group de-index during upload in content-index service Recent performance problem we met for uploading & promotion is all caused by the getAffectedBy operation, which is a time-consumed one for a full scan on store data. After checking, I found that it is mainly used for de-index or re-index content-index entries for the uploading repo. I think this de-index will not be a block factor during the upload or promotion process, which means we could make it async. In this commit, I switched back to use a normal executor service to async the de-index of the affected group. One potential problem for this solution is the executor queue size, which could cause OOM if there are bunch of uploading happened in a short time period. We need to monitor if this can happen. * Code cleanup: remove dupe and unused deps (#1474) Purpose to reduce indy launcher tarball size. Did following: * Removed all duped junit, harmcrest and cdi delaration * Removed quartz deps as it is not used now * Removed h2 as it is not used now * don't build cache migrator or record extractor in main build. (#1478) These don't need to be on the main release cycle, and it seems there's some recursion problem in the maven-assembly-plugin with one of them that's screwing up Indy builds. * Thread-off two nfc cleaning ops which refers getAffectedBy (#1477) * Thread-off getAffectedBy ops for nfc cleaning in ContentManager * Thread-off getAffectedBy ops for nfc cleaning in promote validation * register NFC cache listener in DefaultContentIndexManager (#1479) * Limit "affected-by" with full scan opeartion in package and type based set (#1471) Add a new Cache which is <package, Map<type, Set<StoreKey>> to store all Store keys based on package and type, so for affected group get method, we can avoid the full scan on all repos to get groups, but just simplified to two map get ops. * Fix NPE when store is null in case of being deleted * Thread-off getAffectedBy ops in NFCContentListener * Extract getGroupsAffectedBy from Query to StoreDataManager * Add reverse map for affected stores by some constituents change * Limit "affected-by" with full scan opeartion in package and type based set (#1471) Add a new Cache which is <package, Map<type, Set<StoreKey>> to store all Store keys based on package and type, so for affected group get method, we can avoid the full scan on all repos to get groups, but just simplified to two map get ops. * Simplify affected-by cache We won't completely flatten out the affected-by cache by resolving groups affected by groups affected by a given store. Instead, we leave it as a simple reverse map of the group's constituents, which will give us direct keys to lookup in order to iterate and calculated these n-level affected relationships. It's still much better than scanning all groups (which in the past meant scanning all repos). At the same time, it will avoid any unforseen problems with group membership changes that don't get propagated via the flattening process. It should be easier to maintain. * populate the affected-by cache on startup if the affected-by cache is clear This should work if the affected-by cache is kept in memory, or for the first run when we use a database to store it. We're lining this up with the by-package-type cache as a single-node fix, where we will still need a more scalable solution from a proper database / data design approach before we're ready for multi-node usage. * clean up affected-by iteration and consolidate iteration in default store query I added a null check and disabled-group check to the iteration for finding groups affected by a particular store. The rest of these are just adjusting away from streams and toward intermediary Set instances...things like that. * fix merge error in StoreDataManager * fix remaining compilation errors left from merging * fix NPE when ssl validation repo is null due to some error * Fix potential NPE by using PackageMetadataMerger as a component, as intended. * Update copyright year range to include 2020, reformat license headers (#1484) * fix license headers * Update project copyright year range to include 2020 and reformat license headers * [maven-release-plugin] prepare release indy-parent-1.9.8 * [maven-release-plugin] prepare for next development iteration * Fix some code smell for PromotionManager like potential NPE Conflicts: api/src/main/java/org/commonjava/indy/data/StoreDataManager.java * Some code simplify Conflicts: db/common/src/main/java/org/commonjava/indy/db/common/AbstractStoreDataManager.java * Code optimization - MavenMetadataGenerator * Simplify lambda * Remove useless null check * Use try resource instead * Charset standardization * Add some trace log for metadata clearing of pom update (#1487) * Fix init of affected-by reverse map, and add checks/error message if it's empty (#1488) * Fix affected-by reversemap initialization in lifecycle startup * Add logging to note when the affected-by cache is altered, and another error log message if it's empty * Refactor InfinispanStoreDataManager.init() to startup actions, add some logging The init() method runs before any migration actions run, which means migrated stores won't be stored in the byPkg cache. The old initializer for the affected-by map has a similar problem, so I removed it in favor of the new startup action. The byPkg init was moved out to a startup action too for this reason. Finally, I've added a log message to show which groups were found to be affected by a specific set of input keys. This will give us better information about whether the affected-by map is being used or not. * adding another ftest that initializes store data at boot instead of at migration * Support honeycomb * Fix config * Add HoneycombInterceptor * Fix getchildspan * Add HoneycombFilter * Add pathInfo to fields * Fix NPE in Honeycomb filter and add some fields to honeycomb spans * Enable collection of more metrics (esp. I/O level) via Honeycomb (#1492) This required refactoring the Galley timing function providers, using a new patch of Galley that attempts to generalize the metrics interface, at least for I/O operations. It also involved creating three new interceptors, which would handle methods that are designed to wrap arbitrary pieces of code in metrics. * return the Honeycomb trace span at the filter level, not the root span that is inside it. (#1493) * add golden signals filter into IndyDeployment because filter ordering is important (#1494) * fix cyclic dependency caused by moving GoldenSignalsFilter (#1495) * fix cyclic dependency caused by moving GoldenSignalsFilter * Adding beans.xml to allow CDI injection properly * upgrade galley to release and fix license headers * [maven-release-plugin] prepare release indy-parent-1.9.9 * [maven-release-plugin] prepare for next development iteration * Honeycomb field-capture improvements, and span configuration (#1497) * Fix the Honeycomb root span (which I broke previously) * debug some honeycomb field-tracking problems I had to break out the ThreadContext initialization into an outer servlet filter (specified first in the mapping order, since it's the outer one) to preserve fields until honeycomb can find it. I've also added the ability to turn spans on/off in the configuration, using EITHER include OR exclude semantics (you can't use both). * ignore metrics testing for now...test needs to be updated * Consolidate all async getGroupAffectedBy ops in StoreDataManager * New Relic telemetry support (#1498) * Add New Relic telemetry support, and complete configuration of interceptors in beans.xml for Honeycomb * check before sending batch, push field extraction to manager, use Dequeue instead of List * Add New Relic telemetry support, and complete configuration of interceptors in beans.xml for Honeycomb * check before sending batch, push field extraction to manager, use Dequeue instead of List * Refine / fix new relic span tracking * try to forget we ever used New Relic tracing, the UI is terrible * Add new configuration approach for honeycomb and configurable sampling rates Sampling rates can be specified by Golden Signals (SLI) function name, or by method name, or by class name + method name, etc. * forgot to add configuration parameter for base sample rate * fix compilation error * Tie in threaded-off calls to Honeycomb trace by simulating jump to another service * refactored thread contextualizer for honeycomb I'm ignoring the client bound to other threads and just creating a new one from config when needed. This should work around NPEs in deeper / uncontrolled threads. * Align version to 1.9.10-SNAPSHOT * Update license head * Fix merge issues * Fix merge issues-2 * Fix 2 ftests where the bytestream is drained after the first use * Revert "Fix 2 ftests where the bytestream is drained after the first use" This reverts commit 81df08a2ab3a38d066b65be8b888bbf739c4c223. Co-authored-by: Wenjie Guo <guowj12@gmail.com> Co-authored-by: georgy <3187346+gorgija@users.noreply.github.com> Co-authored-by: Gang Li <ligangty@gmail.com> Co-authored-by: John Casey <jdcasey@commonjava.org> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: geored <47973913+geored@users.noreply.github.com> * Fix merge issue * Solve conflict * Solve conflict * Not delete read-only legacy files * Use constants Co-authored-by: Gang Li <ligangty@gmail.com> Co-authored-by: Wenjie Guo <guowj12@gmail.com> Co-authored-by: georgy <3187346+gorgija@users.noreply.github.com> Co-authored-by: John Casey <jdcasey@commonjava.org> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: geored <47973913+geored@users.noreply.github.com>

Fix groovy.lang.MissingPropertyException: No such property: tarball_url

64f96c9

ruhan1 merged commit 117c780 into Commonjava:master Nov 18, 2019

ruhan1 deleted the master-fix-tarball_url branch November 18, 2019 03:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix groovy.lang.MissingPropertyException: No such property: tarball_url#1428

Fix groovy.lang.MissingPropertyException: No such property: tarball_url#1428
ruhan1 merged 1 commit intoCommonjava:masterfrom
ruhan1:master-fix-tarball_url

ruhan1 commented Nov 18, 2019

Uh oh!

ruhan1 commented Nov 18, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ruhan1 commented Nov 18, 2019

Uh oh!

ruhan1 commented Nov 18, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant