Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Publish 'artifacts.properties' using matrix params #6014

Merged
merged 34 commits into from Dec 2, 2019

Conversation

@jgsogo
Copy link
Member

jgsogo commented Oct 31, 2019

Changelog: Feature: Information in the artifacts.properties file is sent using matrix-params too when a package is uploaded to a server (if it has the capability). This will be the recommended way to send these properties to Artifactory (release TBD) to bypass Nginx blocking properties with periods.
Docs: conan-io/docs#1487

On top of #6013

  • TODO: URL-encode matrix parameters
  • Conan-Server ignores all this key-value pairs
  • TODO: if we want to send a list of values for the same property, there are two alternatives:
    • Use key=value1;value2 in the artifacts.property file.
    • Modify the representation of the artifacts.properties in the sources to allow duplicated values (see patch here)
  • TODO: do we want to add this capability to ConanServer?
  • conan-server doesn't support values with character / (Artifactory does). Maybe using a special filter for the matrix_params part of the URL it will. conan-server doesn't accept matrix_params (they are sent if the server returns a matrix_params capability)
  • Do we want an opt-in? Using matrix_params capability

#REVISIONS: 1
#TAGS: slow

@jgsogo jgsogo self-assigned this Oct 31, 2019
@jgsogo jgsogo force-pushed the jgsogo:feature/publish-bi-matrix-params branch 2 times, most recently from 770ce67 to 96cb4f0 Nov 4, 2019
@jgsogo jgsogo force-pushed the jgsogo:feature/publish-bi-matrix-params branch from 96cb4f0 to 40e9b4e Nov 4, 2019
@lasote lasote added this to the 1.21 milestone Nov 5, 2019
@jgsogo jgsogo marked this pull request as ready for review Nov 5, 2019
@jgsogo jgsogo changed the title [WIP] Publish 'artifacts.properties' using matrix params Publish 'artifacts.properties' using matrix params Nov 7, 2019
@lasote lasote assigned lasote and unassigned jgsogo Nov 7, 2019
Copy link
Contributor

lasote left a comment

I think I would try to use the CHECKSUM_DEPLOY capability to recognize Artifactory and use that as an opt-in. All the changes in conan-server could be dropped. Could you take a look and consider?

self.base_url = base_url

if artifacts_properties:
matrix_params = dict(_remove_put_prefix(artifacts_properties))

This comment has been minimized.

Copy link
@lasote

lasote Nov 7, 2019

Contributor

What would happen if we send both?

This comment has been minimized.

Copy link
@jgsogo

jgsogo Nov 7, 2019

Author Member

Artifactory adds all the properties in this order: (1) headers, (2) files and (3) matrix params. I think, but I need to check (I'll add it to a TODO list), that duplicated items are not stored/shown.

@jgsogo

This comment has been minimized.

Copy link
Member Author

jgsogo commented Nov 7, 2019

If we are going to use a capability I would use a custom one to keep it easy for other servers in the wild: they might be already using CHECKSUM_DEPLOY for a different thing and they might not be prepared to receive matrix params. But that decision is out of my hands.

@jgsogo jgsogo self-assigned this Nov 7, 2019
@lasote

This comment has been minimized.

Copy link
Contributor

lasote commented Nov 7, 2019

If they might not be prepared to receive matrix params your are breaking them all in this PR

@jgsogo

This comment has been minimized.

Copy link
Member Author

jgsogo commented Nov 7, 2019

That's the reason I suggested an opt-in, but one dedicated.

@lasote

This comment has been minimized.

Copy link
Contributor

lasote commented Nov 7, 2019

Mhhh. But I'm thinking the optin cannot be in the client, we need a new capability anyway, especially if we cannot send the properties by the old and the new mechanism. Given that the implementation in Artifactory has been necessary, introduce a "matrix_properties" capability makes sense. We can get rid of all the changes in the server, we won't break other servers and we can send only by one mechanism.

Copy link
Contributor

lasote left a comment

Resolve the little conflict also.

conans/test/functional/remote/put_properties_test.py Outdated Show resolved Hide resolved
conans/test/functional/remote/put_properties_test.py Outdated Show resolved Hide resolved
conans/client/rest/rest_client_v1.py Outdated Show resolved Hide resolved
@jgsogo jgsogo requested a review from lasote Nov 25, 2019
@lasote
lasote approved these changes Dec 2, 2019
Copy link
Contributor

lasote left a comment

Complete the changelog. Probably not needed docs but take a look just in case it deserves some note or clarification.

@lasote lasote merged commit 8110243 into conan-io:develop Dec 2, 2019
2 checks passed
2 checks passed
continuous-integration/jenkins/pr-head This commit looks good
Details
license/cla Contributor License Agreement is signed.
Details
@jgsogo jgsogo deleted the jgsogo:feature/publish-bi-matrix-params branch Dec 2, 2019
memsharded added a commit to memsharded/conan that referenced this pull request Dec 3, 2019
* draft to test

* add missing argument to functions

* rename 'put_headers' to 'artifact_properties'

* store matrix params as the preformatted string in the router

* fallback to empty string

* something going wrong: too many cchanges

* at least these are not needed

* typo

* it was only needed to remove the matrix_params for the bottle server

* soooo

* remove class name

* fix minor bug

* now with revisions too

* make docs with triple double quotes

* we need to modify the conan_server to accept matrix_params too

* fix tests

* add tests with artifacts.properties

* quote values of matrix params

* we can stack routes

* add tests with many different values

* remove changes not needed

* use server capability to choose the URL to build

* do not send headers if matrix_params capability

* do not add 'matrix_params' functionality to conan-server

* rename variable, 'file' is reserved

* remove change uneeded

* conan server does not support matrix params at all

* use the proper if/else
lasote added a commit that referenced this pull request Dec 3, 2019
* removing pylint from conan codebase. TODO: Move to hooks

* add python_requires to 'conans'

* Fix SyntaxWarning in Python 3.8 (#6165)

* - generate ConfigVersion.cmake file (#6063)

Signed-off-by: SSE4 <tomskside@gmail.com>

* Fixes #5925 Make the first line from find program as the result (#6039)

* 3. extra separator in Windows

* Improve the error message when failed to connect to remote

* Improve the error message when failed to connect to remote

* Improve the error message when failed to connect to remote

* Revert unwanted changes

* Pick the first match which is the best match

* Pick the first match which is the best match

* make reversed explicit (better that [::-1])

* Fix output folder for conanworkspace.cmake when rebuilding dependencies (#6060)

* Added required = True to subparsers in order to print error message in Py2 and Py3.

* sync

* basic concurrent upload at reference level with futures

* revert changes

* add line

* Lock buggy urllib3 (#5808)

* app simplifying (#5806)

* Apply lockfile before updating downstream requires (#5771)

* apply graph_lock before looking for overrides

* first step: get rid of the warning

* cleaner if graph_lock is passed to the function

* only update requires upstream if no lockfile is applied

* fix tests

* Deprecation of CONAN_USERNAME and CONAN_CHANNEL: fix error message (#5756)

* if CONAN_USERNAME and CONAN_CHANNEL are deprecated, the error cannot recommend them

* update tests accordingly

* test client load() file method (#5815)

* no user/channel repr without _ (#5817)

* no user/channel repr without _

* minor fixes

* fix tests

* Remove py34 (#5820)

* fix upload package id (#5824)

* - update macOS, watchOS, tvOS, iOS version numbers (#5823)

* Refresh token client support.  (#5662)

* Refresh token client support. Missing tests. Missing migration

* public method

* WIP

* Refresh almost there

* Removed prints

* Try migrate

* Migration

* Add comment

* Refresh token flow following RFC recommentations

* Refresh ok

* review

* Remove traces

* Refactor capabilities

* Removed tmp file

* Review

* #5819 Show warning message for Python 3.4 (#5829)

* #5819 Show warning message for Python 3.4

- Add new warning message for python 3.4 which is no longer supported
- Added funcional tests to validate both python 3.4 and 2.x

Signed-off-by: Uilian Ries <uilianries@gmail.com>

* #5819 Fix broken tests

Signed-off-by: Uilian Ries <uilianries@gmail.com>

* Add cpp_info.name to cmake and pkg_config generators (#5598)

* Add cpp_info.name to cmake generators

* Fix unit tests to mimic real behavior

* cmake_paths test

* add test for cmake generator

* Add cmake_find_package test

* fix test in py3

* Applied cpp_info.name to pkg_config generator

* check different name in pkg_config

* sync with develop

* save conanworkspace.cmake relative to base folder

* add test

* siplify test

* add context manager to build folder

* change indent

* Publish 'artifacts.properties' using matrix params (#6014)

* draft to test

* add missing argument to functions

* rename 'put_headers' to 'artifact_properties'

* store matrix params as the preformatted string in the router

* fallback to empty string

* something going wrong: too many cchanges

* at least these are not needed

* typo

* it was only needed to remove the matrix_params for the bottle server

* soooo

* remove class name

* fix minor bug

* now with revisions too

* make docs with triple double quotes

* we need to modify the conan_server to accept matrix_params too

* fix tests

* add tests with artifacts.properties

* quote values of matrix params

* we can stack routes

* add tests with many different values

* remove changes not needed

* use server capability to choose the URL to build

* do not send headers if matrix_params capability

* do not add 'matrix_params' functionality to conan-server

* rename variable, 'file' is reserved

* remove change uneeded

* conan server does not support matrix params at all

* use the proper if/else

* Add res dir to the variables when using cmake_find_package. (issue 3722) (#6166)

* skip downloading call to to_file_bytes (#6142)

* warn if env-vars defined

* removed warning and env-vars

* remove another env-var
memsharded added a commit to memsharded/conan that referenced this pull request Dec 3, 2019
* draft to test

* add missing argument to functions

* rename 'put_headers' to 'artifact_properties'

* store matrix params as the preformatted string in the router

* fallback to empty string

* something going wrong: too many cchanges

* at least these are not needed

* typo

* it was only needed to remove the matrix_params for the bottle server

* soooo

* remove class name

* fix minor bug

* now with revisions too

* make docs with triple double quotes

* we need to modify the conan_server to accept matrix_params too

* fix tests

* add tests with artifacts.properties

* quote values of matrix params

* we can stack routes

* add tests with many different values

* remove changes not needed

* use server capability to choose the URL to build

* do not send headers if matrix_params capability

* do not add 'matrix_params' functionality to conan-server

* rename variable, 'file' is reserved

* remove change uneeded

* conan server does not support matrix params at all

* use the proper if/else
lasote added a commit that referenced this pull request Dec 3, 2019
* compatible_package build=missing

* move CONAN_EXPORTED and CONAN_IN_LOCAL_CACHE outside cmake_flags

* do not extract build_type and in_local_cache

* it is moved to the end of the command line

* Change help message for --build policy for create command (#6131)

* change build policy for create

* change to package name

* remove unused imports

* client.load() (#6140)

* Fixes #6044 Improve error with malformed settings yml (#6059)

* Fixes #6044 Improve error raised when 'settings.yml' cannot be parsed

* add tests for invalid settings YAML file

* Fix several issues with download command and revisions (#6138)

* fail if specifying rev with revs disabled

* fix reference with revisions

* add package revision to id

* add tests

* check revisions

* fix create without user channel

* add test without user channel

* minor changes

* skip depending on revisions

* add test with fake revision

* remove server from test

* test for older revision in server

* divide test

* Fix upload behaviour when using revisions for packages (#6143)

* wip

* fail if trying to upload different revision than in cache

* fail if rev specified without revs enabled

* add tests

* change line format

* fix python 27

* minor changes

* Fix SyntaxWarning in Python 3.8 (#6165)

* - generate ConfigVersion.cmake file (#6063)

Signed-off-by: SSE4 <tomskside@gmail.com>

* Fixes #5925 Make the first line from find program as the result (#6039)

* 3. extra separator in Windows

* Improve the error message when failed to connect to remote

* Improve the error message when failed to connect to remote

* Improve the error message when failed to connect to remote

* Revert unwanted changes

* Pick the first match which is the best match

* Pick the first match which is the best match

* make reversed explicit (better that [::-1])

* Fix output folder for conanworkspace.cmake when rebuilding dependencies (#6060)

* Added required = True to subparsers in order to print error message in Py2 and Py3.

* sync

* basic concurrent upload at reference level with futures

* revert changes

* add line

* Lock buggy urllib3 (#5808)

* app simplifying (#5806)

* Apply lockfile before updating downstream requires (#5771)

* apply graph_lock before looking for overrides

* first step: get rid of the warning

* cleaner if graph_lock is passed to the function

* only update requires upstream if no lockfile is applied

* fix tests

* Deprecation of CONAN_USERNAME and CONAN_CHANNEL: fix error message (#5756)

* if CONAN_USERNAME and CONAN_CHANNEL are deprecated, the error cannot recommend them

* update tests accordingly

* test client load() file method (#5815)

* no user/channel repr without _ (#5817)

* no user/channel repr without _

* minor fixes

* fix tests

* Remove py34 (#5820)

* fix upload package id (#5824)

* - update macOS, watchOS, tvOS, iOS version numbers (#5823)

* Refresh token client support.  (#5662)

* Refresh token client support. Missing tests. Missing migration

* public method

* WIP

* Refresh almost there

* Removed prints

* Try migrate

* Migration

* Add comment

* Refresh token flow following RFC recommentations

* Refresh ok

* review

* Remove traces

* Refactor capabilities

* Removed tmp file

* Review

* #5819 Show warning message for Python 3.4 (#5829)

* #5819 Show warning message for Python 3.4

- Add new warning message for python 3.4 which is no longer supported
- Added funcional tests to validate both python 3.4 and 2.x

Signed-off-by: Uilian Ries <uilianries@gmail.com>

* #5819 Fix broken tests

Signed-off-by: Uilian Ries <uilianries@gmail.com>

* Add cpp_info.name to cmake and pkg_config generators (#5598)

* Add cpp_info.name to cmake generators

* Fix unit tests to mimic real behavior

* cmake_paths test

* add test for cmake generator

* Add cmake_find_package test

* fix test in py3

* Applied cpp_info.name to pkg_config generator

* check different name in pkg_config

* sync with develop

* save conanworkspace.cmake relative to base folder

* add test

* siplify test

* add context manager to build folder

* change indent

* Publish 'artifacts.properties' using matrix params (#6014)

* draft to test

* add missing argument to functions

* rename 'put_headers' to 'artifact_properties'

* store matrix params as the preformatted string in the router

* fallback to empty string

* something going wrong: too many cchanges

* at least these are not needed

* typo

* it was only needed to remove the matrix_params for the bottle server

* soooo

* remove class name

* fix minor bug

* now with revisions too

* make docs with triple double quotes

* we need to modify the conan_server to accept matrix_params too

* fix tests

* add tests with artifacts.properties

* quote values of matrix params

* we can stack routes

* add tests with many different values

* remove changes not needed

* use server capability to choose the URL to build

* do not send headers if matrix_params capability

* do not add 'matrix_params' functionality to conan-server

* rename variable, 'file' is reserved

* remove change uneeded

* conan server does not support matrix params at all

* use the proper if/else

* Add res dir to the variables when using cmake_find_package. (issue 3722) (#6166)

* skip downloading call to to_file_bytes (#6142)

* remove syntax warning (#6174)

* fix cwd issues in set_version() evaluation (#6130)

* fix cwd issues in set_version() evaluation

* explicit self.recipe_folder
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.