Strange behavior of Artifactory remote repos #5396
Comments
|
Thanks for the info, I'm asking Artifactory support for help with this. |
|
JIRA issue: https://www.jfrog.com/jira/browse/RTFACT-19475 @db4 it would be useful if you could add the information of which Artifactory versions, and a complete trace (showing that a second attempt to download will re-download from the proxied repo, that trace could be like the first time those packages are hit). A couple of lines how to reproduce with the conan client (including version) could also be good. Thanks! |
Artifactory 6.10.4 CE for C/C++
Well, it's the complete trace. There are more packages but for all of them, Artifactory does the same. For any package in the local cache, it requests
I use the latest conan client (1.16.1)/conan package tools 0.28.1 with Gitlab CI. But I believe |
|
Have you checked the "store artifacts locally" on the remote repository? The team is not able to reproduce the issue. If it is found locally it shouldn't go remotely. |
yes
Even to find revisions available on the remote? The problem is not artifact downloading but discovering remote revisions for cached artifacts. It initiates a network connection (see the log above) and introduces significant delay. |
|
Hello, The client will search the recipe in the first remote since it is not found he will search in the second remote and found it. On my end, I successfully install "boost_test/1.66.0@bincrafters/stable" after adding the above remote and this flag " --build "boost_*" ", i.e.: Hope this helps. |
See my first message. Of course, I've created two Artifactory remote repos (for conan-center and bincrafters) |
|
I would like to reproduce this behavior on my end in order to debug this issue, to do that, I need the build a similar setup to you, for this, I need the config descriptor file located in {ARTIFACTORY_HOME}/etc/artifactory.config.latest.xml, and the output of the command: Waiting forward to your reply. |
Do you need the full file? If yes, please let me know how to send it privately. <remoteRepositories>
<remoteRepository>
<key>bincrafters</key>
<type>conan</type>
<includesPattern>**/*</includesPattern>
<repoLayoutRef>conan-default</repoLayoutRef>
<dockerApiVersion>V2</dockerApiVersion>
<forceNugetAuthentication>false</forceNugetAuthentication>
<blackedOut>false</blackedOut>
<handleReleases>true</handleReleases>
<handleSnapshots>true</handleSnapshots>
<maxUniqueSnapshots>0</maxUniqueSnapshots>
<maxUniqueTags>0</maxUniqueTags>
<suppressPomConsistencyChecks>true</suppressPomConsistencyChecks>
<propertySets/>
<archiveBrowsingEnabled>false</archiveBrowsingEnabled>
<url>https://api.bintray.com/conan/bincrafters/public-conan</url>
<offline>false</offline>
<hardFail>false</hardFail>
<storeArtifactsLocally>true</storeArtifactsLocally>
<fetchJarsEagerly>false</fetchJarsEagerly>
<fetchSourcesEagerly>false</fetchSourcesEagerly>
<retrievalCachePeriodSecs>600</retrievalCachePeriodSecs>
<assumedOfflinePeriodSecs>300</assumedOfflinePeriodSecs>
<missedRetrievalCachePeriodSecs>1800</missedRetrievalCachePeriodSecs>
<remoteRepoChecksumPolicyType>generate-if-absent</remoteRepoChecksumPolicyType>
<unusedArtifactsCleanupPeriodHours>0</unusedArtifactsCleanupPeriodHours>
<shareConfiguration>false</shareConfiguration>
<synchronizeProperties>false</synchronizeProperties>
<listRemoteFolderItems>true</listRemoteFolderItems>
<rejectInvalidJars>false</rejectInvalidJars>
<contentSynchronisation>
<enabled>false</enabled>
<statistics>
<enabled>false</enabled>
</statistics>
<properties>
<enabled>false</enabled>
</properties>
<source>
<originAbsenceDetection>false</originAbsenceDetection>
</source>
</contentSynchronisation>
<blockMismatchingMimeTypes>true</blockMismatchingMimeTypes>
<mismatchingMimeTypesOverrideList></mismatchingMimeTypesOverrideList>
<bypassHeadRequests>false</bypassHeadRequests>
<allowAnyHostAuth>false</allowAnyHostAuth>
<socketTimeoutMillis>15000</socketTimeoutMillis>
<enableCookieManagement>false</enableCookieManagement>
<enableTokenAuthentication>false</enableTokenAuthentication>
<propagateQueryParams>false</propagateQueryParams>
</remoteRepository>
<remoteRepository>
<key>conan-center</key>
<type>conan</type>
<includesPattern>**/*</includesPattern>
<repoLayoutRef>conan-default</repoLayoutRef>
<dockerApiVersion>V2</dockerApiVersion>
<forceNugetAuthentication>false</forceNugetAuthentication>
<blackedOut>false</blackedOut>
<handleReleases>true</handleReleases>
<handleSnapshots>true</handleSnapshots>
<maxUniqueSnapshots>0</maxUniqueSnapshots>
<maxUniqueTags>0</maxUniqueTags>
<suppressPomConsistencyChecks>true</suppressPomConsistencyChecks>
<propertySets/>
<archiveBrowsingEnabled>false</archiveBrowsingEnabled>
<url>https://conan.bintray.com</url>
<offline>false</offline>
<hardFail>false</hardFail>
<storeArtifactsLocally>true</storeArtifactsLocally>
<fetchJarsEagerly>false</fetchJarsEagerly>
<fetchSourcesEagerly>false</fetchSourcesEagerly>
<retrievalCachePeriodSecs>600</retrievalCachePeriodSecs>
<assumedOfflinePeriodSecs>300</assumedOfflinePeriodSecs>
<missedRetrievalCachePeriodSecs>1800</missedRetrievalCachePeriodSecs>
<remoteRepoChecksumPolicyType>generate-if-absent</remoteRepoChecksumPolicyType>
<unusedArtifactsCleanupPeriodHours>0</unusedArtifactsCleanupPeriodHours>
<shareConfiguration>false</shareConfiguration>
<synchronizeProperties>false</synchronizeProperties>
<listRemoteFolderItems>true</listRemoteFolderItems>
<rejectInvalidJars>false</rejectInvalidJars>
<contentSynchronisation>
<enabled>false</enabled>
<statistics>
<enabled>false</enabled>
</statistics>
<properties>
<enabled>false</enabled>
</properties>
<source>
<originAbsenceDetection>false</originAbsenceDetection>
</source>
</contentSynchronisation>
<blockMismatchingMimeTypes>true</blockMismatchingMimeTypes>
<mismatchingMimeTypesOverrideList></mismatchingMimeTypesOverrideList>
<bypassHeadRequests>false</bypassHeadRequests>
<allowAnyHostAuth>false</allowAnyHostAuth>
<socketTimeoutMillis>15000</socketTimeoutMillis>
<enableCookieManagement>false</enableCookieManagement>
<enableTokenAuthentication>false</enableTokenAuthentication>
<propagateQueryParams>false</propagateQueryParams>
</remoteRepository>
</remoteRepositories>
Indeed, I have bincrafters repo included twice, an Artifactory-cached repo ( |
|
I think it would be great if we could reduce the scope of this issue and simplify it. First step: Then, a single Thanks for the feedback! |
|
This is not an issue, it is the expected behavior. In the log from the first message: all the requests are looking for the list of revisions ( We can check this behavior if we compare it with the logs from the first time we get a package, it shows more calls to the proxy downloading files like How to avoid this extra call:
Please, let me know if you any doubt. |
Tried that. It only works for direct dependencies. For transitive dependencies, of which there are dozens for modular Boost, it's still calling through to Bincrafters bintray, at a cost of 6-7 seconds per failed retrieval of |
|
The problem with transitive dependencies is that the revision is not indicated for the transitive dependencies (it isn't for modular boost, indeed), so those transitive dependencies are trying to resolve the latest revision. This is a huge issue, indeed, and to improve it we would need to touch many different products: client, servers,... :/ There are some ideas in the backlog that may alleviate this pain: using a lockfile, once the revision for all the dependencies is set, will help with this, but major improvement should probably come from the server side. |
|
Sadly, lockfiles are a solution that creates a whole nest of problems, since I'd have to somehow ship a combinatorial explosion of lockfiles to all my teams (lost of projects on lots of platforms with lots of configurations), and all the lockfiles would be invalid once one of our own dependencies changed. Are there some specific Jira cases I could subscribe to that would at least let me know the progress on these improvements? |
|
I'm not sure if the issue reported in #5432 has the same origin or solution, but it is also about performance related to Artifactory. About the lockfiles and how to handle them within a company, probably @lasote can give you some advice about how to use them (ping him next week), and we would be pleased to know any feedback about them you may have. It is a feature we want to improve and push forward because we think it is very valuable for our users. This is not the issue to talk about lockfiles, better not to mix topics, but feel free to open a new one or you can reach us on Slack too. Thanks! |
Trying to find an interim solution for #5175 I set up remote Artifactory repos for conan-center (https://conan.bintray.com) and bincrafters (https://api.bintray.com/conan/bincrafters/public-conan) conan repos and used them for CI jobs. I expected Artifactory to get packages from its local cache when they are available and don't waste precious time requesting the remote for anything, but Artifactory logs show quite different behavior. Why?
The text was updated successfully, but these errors were encountered: