Publish 'artifacts.properties' using matrix params #6014

jgsogo · 2019-10-31T19:14:17Z

Changelog: Feature: Information in the artifacts.properties file is sent using matrix-params too when a package is uploaded to a server (if it has the capability). This will be the recommended way to send these properties to Artifactory (release TBD) to bypass Nginx blocking properties with periods.
Docs: conan-io/docs#1487

On top of #6013

TODO: URL-encode matrix parameters
Conan-Server ignores all this key-value pairs
TODO: if we want to send a list of values for the same property, there are two alternatives:
- Use key=value1;value2 in the artifacts.property file.
- ~~Modify the representation of the artifacts.properties in the sources to allow duplicated values (see patch here)~~
TODO: do we want to add this capability to ConanServer?
conan-server doesn't support values with character / (Artifactory does). Maybe using a special filter for the matrix_params part of the URL it will. ❎ conan-server doesn't accept matrix_params (they are sent if the server returns a matrix_params capability)
Do we want an opt-in? Using matrix_params capability

#REVISIONS: 1
#TAGS: slow

lasote

I think I would try to use the CHECKSUM_DEPLOY capability to recognize Artifactory and use that as an opt-in. All the changes in conan-server could be dropped. Could you take a look and consider?

lasote · 2019-11-07T12:19:01Z

conans/client/rest/client_routes.py

+        self.base_url = base_url
+
+        if artifacts_properties:
+            matrix_params = dict(_remove_put_prefix(artifacts_properties))


What would happen if we send both?

Artifactory adds all the properties in this order: (1) headers, (2) files and (3) matrix params. I think, but I need to check (I'll add it to a TODO list), that duplicated items are not stored/shown.

jgsogo · 2019-11-07T18:28:45Z

If we are going to use a capability I would use a custom one to keep it easy for other servers in the wild: they might be already using CHECKSUM_DEPLOY for a different thing and they might not be prepared to receive matrix params. But that decision is out of my hands.

lasote · 2019-11-07T19:46:39Z

If they might not be prepared to receive matrix params your are breaking them all in this PR

jgsogo · 2019-11-07T20:02:01Z

That's the reason I suggested an opt-in, but one dedicated.

lasote · 2019-11-07T21:48:01Z

Mhhh. But I'm thinking the optin cannot be in the client, we need a new capability anyway, especially if we cannot send the properties by the old and the new mechanism. Given that the implementation in Artifactory has been necessary, introduce a "matrix_properties" capability makes sense. We can get rid of all the changes in the server, we won't break other servers and we can send only by one mechanism.

lasote

Resolve the little conflict also.

conans/test/functional/remote/put_properties_test.py

conans/client/rest/rest_client_v1.py

lasote

Complete the changelog. Probably not needed docs but take a look just in case it deserves some note or clarification.

* draft to test * add missing argument to functions * rename 'put_headers' to 'artifact_properties' * store matrix params as the preformatted string in the router * fallback to empty string * something going wrong: too many cchanges * at least these are not needed * typo * it was only needed to remove the matrix_params for the bottle server * soooo * remove class name * fix minor bug * now with revisions too * make docs with triple double quotes * we need to modify the conan_server to accept matrix_params too * fix tests * add tests with artifacts.properties * quote values of matrix params * we can stack routes * add tests with many different values * remove changes not needed * use server capability to choose the URL to build * do not send headers if matrix_params capability * do not add 'matrix_params' functionality to conan-server * rename variable, 'file' is reserved * remove change uneeded * conan server does not support matrix params at all * use the proper if/else

* removing pylint from conan codebase. TODO: Move to hooks * add python_requires to 'conans' * Fix SyntaxWarning in Python 3.8 (#6165) * - generate ConfigVersion.cmake file (#6063) Signed-off-by: SSE4 <tomskside@gmail.com> * Fixes #5925 Make the first line from find program as the result (#6039) * 3. extra separator in Windows * Improve the error message when failed to connect to remote * Improve the error message when failed to connect to remote * Improve the error message when failed to connect to remote * Revert unwanted changes * Pick the first match which is the best match * Pick the first match which is the best match * make reversed explicit (better that [::-1]) * Fix output folder for conanworkspace.cmake when rebuilding dependencies (#6060) * Added required = True to subparsers in order to print error message in Py2 and Py3. * sync * basic concurrent upload at reference level with futures * revert changes * add line * Lock buggy urllib3 (#5808) * app simplifying (#5806) * Apply lockfile before updating downstream requires (#5771) * apply graph_lock before looking for overrides * first step: get rid of the warning * cleaner if graph_lock is passed to the function * only update requires upstream if no lockfile is applied * fix tests * Deprecation of CONAN_USERNAME and CONAN_CHANNEL: fix error message (#5756) * if CONAN_USERNAME and CONAN_CHANNEL are deprecated, the error cannot recommend them * update tests accordingly * test client load() file method (#5815) * no user/channel repr without _ (#5817) * no user/channel repr without _ * minor fixes * fix tests * Remove py34 (#5820) * fix upload package id (#5824) * - update macOS, watchOS, tvOS, iOS version numbers (#5823) * Refresh token client support. (#5662) * Refresh token client support. Missing tests. Missing migration * public method * WIP * Refresh almost there * Removed prints * Try migrate * Migration * Add comment * Refresh token flow following RFC recommentations * Refresh ok * review * Remove traces * Refactor capabilities * Removed tmp file * Review * #5819 Show warning message for Python 3.4 (#5829) * #5819 Show warning message for Python 3.4 - Add new warning message for python 3.4 which is no longer supported - Added funcional tests to validate both python 3.4 and 2.x Signed-off-by: Uilian Ries <uilianries@gmail.com> * #5819 Fix broken tests Signed-off-by: Uilian Ries <uilianries@gmail.com> * Add cpp_info.name to cmake and pkg_config generators (#5598) * Add cpp_info.name to cmake generators * Fix unit tests to mimic real behavior * cmake_paths test * add test for cmake generator * Add cmake_find_package test * fix test in py3 * Applied cpp_info.name to pkg_config generator * check different name in pkg_config * sync with develop * save conanworkspace.cmake relative to base folder * add test * siplify test * add context manager to build folder * change indent * Publish 'artifacts.properties' using matrix params (#6014) * draft to test * add missing argument to functions * rename 'put_headers' to 'artifact_properties' * store matrix params as the preformatted string in the router * fallback to empty string * something going wrong: too many cchanges * at least these are not needed * typo * it was only needed to remove the matrix_params for the bottle server * soooo * remove class name * fix minor bug * now with revisions too * make docs with triple double quotes * we need to modify the conan_server to accept matrix_params too * fix tests * add tests with artifacts.properties * quote values of matrix params * we can stack routes * add tests with many different values * remove changes not needed * use server capability to choose the URL to build * do not send headers if matrix_params capability * do not add 'matrix_params' functionality to conan-server * rename variable, 'file' is reserved * remove change uneeded * conan server does not support matrix params at all * use the proper if/else * Add res dir to the variables when using cmake_find_package. (issue 3722) (#6166) * skip downloading call to to_file_bytes (#6142) * warn if env-vars defined * removed warning and env-vars * remove another env-var

* draft to test * add missing argument to functions * rename 'put_headers' to 'artifact_properties' * store matrix params as the preformatted string in the router * fallback to empty string * something going wrong: too many cchanges * at least these are not needed * typo * it was only needed to remove the matrix_params for the bottle server * soooo * remove class name * fix minor bug * now with revisions too * make docs with triple double quotes * we need to modify the conan_server to accept matrix_params too * fix tests * add tests with artifacts.properties * quote values of matrix params * we can stack routes * add tests with many different values * remove changes not needed * use server capability to choose the URL to build * do not send headers if matrix_params capability * do not add 'matrix_params' functionality to conan-server * rename variable, 'file' is reserved * remove change uneeded * conan server does not support matrix params at all * use the proper if/else

* compatible_package build=missing * move CONAN_EXPORTED and CONAN_IN_LOCAL_CACHE outside cmake_flags * do not extract build_type and in_local_cache * it is moved to the end of the command line * Change help message for --build policy for create command (#6131) * change build policy for create * change to package name * remove unused imports * client.load() (#6140) * Fixes #6044 Improve error with malformed settings yml (#6059) * Fixes #6044 Improve error raised when 'settings.yml' cannot be parsed * add tests for invalid settings YAML file * Fix several issues with download command and revisions (#6138) * fail if specifying rev with revs disabled * fix reference with revisions * add package revision to id * add tests * check revisions * fix create without user channel * add test without user channel * minor changes * skip depending on revisions * add test with fake revision * remove server from test * test for older revision in server * divide test * Fix upload behaviour when using revisions for packages (#6143) * wip * fail if trying to upload different revision than in cache * fail if rev specified without revs enabled * add tests * change line format * fix python 27 * minor changes * Fix SyntaxWarning in Python 3.8 (#6165) * - generate ConfigVersion.cmake file (#6063) Signed-off-by: SSE4 <tomskside@gmail.com> * Fixes #5925 Make the first line from find program as the result (#6039) * 3. extra separator in Windows * Improve the error message when failed to connect to remote * Improve the error message when failed to connect to remote * Improve the error message when failed to connect to remote * Revert unwanted changes * Pick the first match which is the best match * Pick the first match which is the best match * make reversed explicit (better that [::-1]) * Fix output folder for conanworkspace.cmake when rebuilding dependencies (#6060) * Added required = True to subparsers in order to print error message in Py2 and Py3. * sync * basic concurrent upload at reference level with futures * revert changes * add line * Lock buggy urllib3 (#5808) * app simplifying (#5806) * Apply lockfile before updating downstream requires (#5771) * apply graph_lock before looking for overrides * first step: get rid of the warning * cleaner if graph_lock is passed to the function * only update requires upstream if no lockfile is applied * fix tests * Deprecation of CONAN_USERNAME and CONAN_CHANNEL: fix error message (#5756) * if CONAN_USERNAME and CONAN_CHANNEL are deprecated, the error cannot recommend them * update tests accordingly * test client load() file method (#5815) * no user/channel repr without _ (#5817) * no user/channel repr without _ * minor fixes * fix tests * Remove py34 (#5820) * fix upload package id (#5824) * - update macOS, watchOS, tvOS, iOS version numbers (#5823) * Refresh token client support. (#5662) * Refresh token client support. Missing tests. Missing migration * public method * WIP * Refresh almost there * Removed prints * Try migrate * Migration * Add comment * Refresh token flow following RFC recommentations * Refresh ok * review * Remove traces * Refactor capabilities * Removed tmp file * Review * #5819 Show warning message for Python 3.4 (#5829) * #5819 Show warning message for Python 3.4 - Add new warning message for python 3.4 which is no longer supported - Added funcional tests to validate both python 3.4 and 2.x Signed-off-by: Uilian Ries <uilianries@gmail.com> * #5819 Fix broken tests Signed-off-by: Uilian Ries <uilianries@gmail.com> * Add cpp_info.name to cmake and pkg_config generators (#5598) * Add cpp_info.name to cmake generators * Fix unit tests to mimic real behavior * cmake_paths test * add test for cmake generator * Add cmake_find_package test * fix test in py3 * Applied cpp_info.name to pkg_config generator * check different name in pkg_config * sync with develop * save conanworkspace.cmake relative to base folder * add test * siplify test * add context manager to build folder * change indent * Publish 'artifacts.properties' using matrix params (#6014) * draft to test * add missing argument to functions * rename 'put_headers' to 'artifact_properties' * store matrix params as the preformatted string in the router * fallback to empty string * something going wrong: too many cchanges * at least these are not needed * typo * it was only needed to remove the matrix_params for the bottle server * soooo * remove class name * fix minor bug * now with revisions too * make docs with triple double quotes * we need to modify the conan_server to accept matrix_params too * fix tests * add tests with artifacts.properties * quote values of matrix params * we can stack routes * add tests with many different values * remove changes not needed * use server capability to choose the URL to build * do not send headers if matrix_params capability * do not add 'matrix_params' functionality to conan-server * rename variable, 'file' is reserved * remove change uneeded * conan server does not support matrix params at all * use the proper if/else * Add res dir to the variables when using cmake_find_package. (issue 3722) (#6166) * skip downloading call to to_file_bytes (#6142) * remove syntax warning (#6174) * fix cwd issues in set_version() evaluation (#6130) * fix cwd issues in set_version() evaluation * explicit self.recipe_folder

jgsogo added 14 commits October 31, 2019 12:02

draft to test

ce6ce71

add missing argument to functions

6fd7c4a

rename 'put_headers' to 'artifact_properties'

f77cfc3

Merge branch 'refact/put_headers' into feature/publish-bi-matrix-params

68fc410

store matrix params as the preformatted string in the router

61f3b57

fallback to empty string

b057442

something going wrong: too many cchanges

7fbc0d3

at least these are not needed

2680c7d

typo

276398c

Merge branch 'refact/put_headers' into feature/publish-bi-matrix-params

09dcf55

it was only needed to remove the matrix_params for the bottle server

3cd74ef

soooo

cc118ba

remove class name

e46537f

fix minor bug

2851ad6

jgsogo self-assigned this Oct 31, 2019

jgsogo added 4 commits October 31, 2019 20:36

now with revisions too

808520d

make docs with triple double quotes

162107a

we need to modify the conan_server to accept matrix_params too

5c1437c

fix tests

f8bc6d1

jgsogo force-pushed the feature/publish-bi-matrix-params branch 2 times, most recently from 770ce67 to 96cb4f0 Compare November 4, 2019 16:03

add tests with artifacts.properties

40e9b4e

jgsogo force-pushed the feature/publish-bi-matrix-params branch from 96cb4f0 to 40e9b4e Compare November 4, 2019 16:04

jgsogo added 5 commits November 4, 2019 18:41

quote values of matrix params

27ddabb

we can stack routes

aa2f56f

add tests with many different values

f51a8c9

Merge branch 'develop' into feature/publish-bi-matrix-params

5443fc4

remove changes not needed

5869043

lasote added this to the 1.21 milestone Nov 5, 2019

jgsogo marked this pull request as ready for review November 5, 2019 11:39

Merge branch 'develop' into feature/publish-bi-matrix-params

1c42cc6

jgsogo changed the title ~~[WIP] Publish 'artifacts.properties' using matrix params~~ Publish 'artifacts.properties' using matrix params Nov 7, 2019

lasote assigned lasote and unassigned jgsogo Nov 7, 2019

lasote requested changes Nov 7, 2019

View reviewed changes

jgsogo self-assigned this Nov 7, 2019

jgsogo added 5 commits November 11, 2019 12:16

use server capability to choose the URL to build

4a89327

do not send headers if matrix_params capability

ca8d05c

do not add 'matrix_params' functionality to conan-server

b4522f9

rename variable, 'file' is reserved

d82cf8a

remove change uneeded

e182f11

lasote requested changes Nov 19, 2019

View reviewed changes

conans/test/functional/remote/put_properties_test.py Outdated Show resolved Hide resolved

conans/test/functional/remote/put_properties_test.py Outdated Show resolved Hide resolved

conans/client/rest/rest_client_v1.py Outdated Show resolved Hide resolved

jgsogo added 3 commits November 25, 2019 14:04

Merge branch 'develop' into feature/publish-bi-matrix-params

ce8a2d3

conan server does not support matrix params at all

986c17b

use the proper if/else

d18abc5

jgsogo requested a review from lasote November 25, 2019 17:12

Merge branch 'develop' into feature/publish-bi-matrix-params

9020256

lasote approved these changes Dec 2, 2019

View reviewed changes

lasote merged commit 8110243 into conan-io:develop Dec 2, 2019

jgsogo deleted the feature/publish-bi-matrix-params branch December 2, 2019 17:00

jgsogo mentioned this pull request Dec 2, 2019

Add information about matrix-params in the artifacts.properties reference conan-io/docs#1487

Merged

jgsogo mentioned this pull request Apr 28, 2020

artifactory_property_vcs.url causes strange layout at Artifactory 7.4 jfrog/jenkins-artifactory-plugin#272

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Publish 'artifacts.properties' using matrix params #6014

Publish 'artifacts.properties' using matrix params #6014

jgsogo commented Oct 31, 2019 •

edited

lasote left a comment

lasote Nov 7, 2019

jgsogo Nov 7, 2019

jgsogo commented Nov 7, 2019

lasote commented Nov 7, 2019

jgsogo commented Nov 7, 2019

lasote commented Nov 7, 2019

lasote left a comment

lasote left a comment

Publish 'artifacts.properties' using matrix params #6014

Publish 'artifacts.properties' using matrix params #6014

Conversation

jgsogo commented Oct 31, 2019 • edited

lasote left a comment

Choose a reason for hiding this comment

lasote Nov 7, 2019

Choose a reason for hiding this comment

jgsogo Nov 7, 2019

Choose a reason for hiding this comment

jgsogo commented Nov 7, 2019

lasote commented Nov 7, 2019

jgsogo commented Nov 7, 2019

lasote commented Nov 7, 2019

lasote left a comment

Choose a reason for hiding this comment

lasote left a comment

Choose a reason for hiding this comment

jgsogo commented Oct 31, 2019 •

edited