Conan plugins #3555

danimtb · 2018-09-14T11:35:02Z

Refer to the issue that supports this Pull Request: Related to Authentication of downloaded binaries #480, Package signing #773, [Binary Linters] Check Runtime #1115, [question][enhancement] check open source licenses using spdx-lookup #2342
If the issue has missing info, explain the purpose/use case/pain/need that covers this Pull Request.
I've read the Contributing guide.
I've followed the PEP8 style guides for Python code: Actually some PEP8 compromised for test readability
I've opened another PR in the Conan docs repo to the develop branch, documenting this one. Also adding a description of the changes in the changelog.rst file. https://github.com/conan-io/docs

Plugins are stored ~/.conan/plugins folder. They can be activated in the conan.conf via general.plugins and CONAN_PLUGINS both accepting a list.

This includes a default activated plugin called attribute_checker that includes the basic checks of url/description/license (Removed from code).

The available methods for a plugin to define are:

def pre_export(output, conanfile, conanfile_path, reference, **kwargs):

def post_export(output, conanfile, conanfile_path, reference, **kwargs):

def pre_source(output, conanfile, conanfile_path, **kwargs):

def post_source(output, conanfile, conanfile_path, **kwargs):

def pre_build(output, conanfile, **kwargs):

def post_build(output, conanfile, **kwargs):

def pre_package(output, conanfile, conanfile_path, **kwargs):

def post_package(output, conanfile, conanfile_path, **kwargs):

def pre_upload(output, conanfile_path, reference, remote, **kwargs):

def post_upload(output, conanfile_path, reference, remote, **kwargs):

def pre_upload_recipe(output, conanfile_path, reference, remote, **kwargs):

def post_upload_recipe(output, conanfile_path, reference, remote, **kwargs):

def pre_upload_package(output, conanfile_path, reference, package_id, remote, **kwargs):

def post_upload_package(output, conanfile_path, reference, package_id, remote, **kwargs):

def pre_download(output, reference, remote, **kwargs):

def post_download(output, conanfile_path, reference, remote, **kwargs):

def pre_download_recipe(output, reference, remote, **kwargs):

def post_download_recipe(output, conanfile_path, reference, remote, **kwargs):

def pre_download_package(output, conanfile_path, reference, package_id, remote, **kwargs):

def post_download_package(output, conanfile_path, reference, package_id, remote, **kwargs):

Changelog: Feature: Experimental support for Conan plugins.

jgsogo · 2018-09-17T14:44:59Z

conans/client/packager.py

+        plugin_manager.execute("pre_package", conanfile=conanfile, conanfile_path=conanfile_path,
+                               reference=str(reference), package_id=pkg_id)
+
+        package_output = ScopedOutput("%s package()" % output.scope, output)


This package_output is not being used!

Yes, it is only used for the copy report some lines below

uupss! 😓

lasote · 2018-09-20T13:38:32Z

conans/client/client_cache.py

+    def plugins(self):
+        """Returns a list of plugins inside the plugins folder"""
+        plugins = []
+        for plugin_name in os.listdir(self.plugins_path):


if os.path.isfile(plugin_name)

Also, don't forget to mention in the docs that any py file there will be considered a plugin, and suggest subfolders.

lasote · 2018-09-20T13:56:54Z

conans/client/remote_manager.py

@@ -188,6 +204,10 @@ def get_recipe(self, conan_reference, remote):
        # Make sure that the source dir is deleted
        rm_conandir(self._client_cache.source(conan_reference))
        touch_folder(dest_folder)
+        conanfile_path = self._client_cache.conanfile(conan_reference)
+        self._plugin_manager.execute("post_download_recipe", conanfile_path=conanfile_path,
+                                     reference=str(conan_reference),


i'm not sure.

If passing a conanfile object, something in the conans namespace

Well, the conanfile is something in the userspace too, is the self within a recipe.
But I don't know, I agree that if they are going to use .name, .version, .user, .channel is useful to pass (and document, of course) a ConanFileReference.

On the other hand, if you receive a ConanFileReference you have to know that it is not comparable with a string directly nor formatted in a string without casting to str.
You could always ref = ConanFileReference.loads(ref).

But yeah, probably people will be going to check the channels and things like that, so let's pass references, also for future revisions, not included in a regular str(ref) but a ref.full_reference(). The str conversion/formatting and object in general HAS TO BE well documented in the plugin section.

lasote · 2018-09-20T14:02:36Z

conans/client/conf/__init__.py

@@ -147,7 +149,7 @@
 class ConanClientConfigParser(ConfigParser, object):

    def __init__(self, filename):
-        ConfigParser.__init__(self)
+        ConfigParser.__init__(self, allow_no_value=True)


allow_no_value not used, right?

I think it is actually needed for conan config set plugins.my_plugin as the normal behaviour would be conan config set plugins.my_plugin=plugin_value_if_something

Yes, that parameter is needed

lasote · 2018-09-20T14:12:30Z

conans/test/command/config_test.py

@@ -41,10 +41,6 @@ def errors_test(self):
        self.assertIn("You can't set a full section, please specify a key=value",
                      self.client.user_io.out)

-        error = self.client.run('config set proxies.http:Value', ignore_error=True)


It is a bit dirty but I would prefer even check if the property being set belongs to "plugins" and allow only in that case, recovering the test. I don't like to deteriorate the current behavior.

memsharded · 2018-09-21T12:04:34Z

@danimtb for the documentation, do not document ConanFileReference. Document that what is passed to the plugins is an object with .name, .version and the str() method converts it to string. So we can decouple it in the future.

lasote · 2018-09-21T12:52:51Z

Resolve conflicts!

memsharded

I have a question: I want to run a "binary package quality" check plugin, to both the packages that I am building, but also to the packages I am retrieving from remote servers. The same applies to recipes, what if I want to check something in every recipe that is either exported into the cache, or it is downloaded into the cache from a remote.

memsharded · 2018-09-21T16:53:23Z

conans/client/cmd/export.py

@@ -59,6 +57,10 @@ def cmd_export(conanfile_path, conanfile, reference, keep_source, output, client
    with client_cache.conanfile_write_lock(reference):
        _export_conanfile(conanfile_path, conanfile.output, client_cache, conanfile, reference,
                          keep_source)
+    conanfile_cache_path = client_cache.conanfile(reference)
+    assert os.path.exists(conanfile_cache_path)


assert can be removed

danimtb · 2018-09-24T07:03:07Z

You can make those checks in post_package() when creating those packages locally and in post_download_package() when downloading them from remotes. For the first one you would have the conanfile.package_folder to inspect the binary files, for the second one I guess you would need to compound the path to the package folder using the conanfile_path argument.

lasote · 2018-09-24T07:35:42Z

I think it should work for the use case you mentioned @memsharded. But with the real usage, we will polish it (experimental). We have to reserve some time to implement some real plugin like the ones we want for conan-center or a cool binary linter for libraries. Do you see any architectural issue right now @memsharded ?

memsharded · 2018-09-24T12:56:24Z

Yeah, agree, can be iterated.

Architecture is good, lets merge this!

solvingj · 2018-10-09T16:21:37Z

Where does one go for continued discussion on this feature?

memsharded · 2018-10-09T17:05:44Z

@solvingj

Open a new issue with [plugin] in the title. We also have a "plugin" label for them. There is also the https://github.com/conan-io/plugins repo, for discussions, PR about specific plugins, could be done there, I guess.

jgsogo · 2018-10-09T17:38:12Z

👍 to move the discussion about plugins themselves to the new repo.

Some issues labeled as plugins should be moved there as they are more a plugin request than an issue related to the plugins architecture. I think that @danimtb and I can take care of this task.

* plugin loader * Added plugin folder and conf * POC linter plugin * Execute method of all plugins * PluginManager with initialize and execute * moved output to pluginmanager and use conanfile path * scoped output and removed reference * added recipe plugin and moved to plugin module * default plugin creation and deinitilize * Initial tests * initialize output * plugin default initial test * Completed default test and fixed PY2 import module * remove execfile * added plugin manager to install process * check parameters for each method * fixed tests * Removed ConanPlugin and changed to execute * remove plugins load from api wrapper * fixed plugins basic tests * minimal complete test * complete test and moved plugin manager to remote manager * moved plugin manager to cmd_export * renamed plugins file and check output exception * use conanfile cache path for post_export * finish check path * fix some tests * removed file and fixed some tests * remove changes in command * removed asserts from test * partial review * review plugins and added import one * review: lazy default plugins creation, valid methods, optimization & [plugins] * review 2 * Revert test and file check * used ConanFileReference * remove load import * removed assert

danimtb added 30 commits July 23, 2018 18:25

plugin loader

98c8c02

Added plugin folder and conf

7694051

Merge branch 'develop' into feature/plugins_linters

333a05a

POC linter plugin

a852f01

Merge branch 'develop' into feature/plugins_linters

9382e71

Execute method of all plugins

2c52162

Merge branch 'develop' into feature/plugins_linters

1ad94c9

PluginManager with initialize and execute

9c27a0e

moved output to pluginmanager and use conanfile path

0af6581

Merge branch 'develop' into feature/plugins_linters

1f556b1

scoped output and removed reference

105dbe1

added recipe plugin and moved to plugin module

5201f2e

default plugin creation and deinitilize

4da4d2f

Initial tests

92d5ba4

initialize output

9bcb5ae

plugin default initial test

c0c327c

Completed default test and fixed PY2 import module

6673320

remove execfile

858613e

added plugin manager to install process

0c3a4e6

Merge branch 'develop' into feature/plugins_linters

bf68cae

check parameters for each method

2002991

fixed tests

3a5d7e3

Removed ConanPlugin and changed to execute

0ca0b6b

remove plugins load from api wrapper

4418400

fixed plugins basic tests

9847f97

minimal complete test

7600a01

complete test and moved plugin manager to remote manager

6ce5e27

moved plugin manager to cmd_export

c1a8eb0

renamed plugins file and check output exception

ef1866a

use conanfile cache path for post_export

7b29f7f

review 2

aa03186

jgsogo suggested changes Sep 17, 2018

View reviewed changes

danimtb assigned lasote and unassigned danimtb Sep 19, 2018

jgsogo approved these changes Sep 19, 2018

View reviewed changes

lasote requested changes Sep 20, 2018

View reviewed changes

Revert test and file check

790626a

ghost assigned danimtb Sep 20, 2018

danimtb added 2 commits September 20, 2018 18:58

used ConanFileReference

8763fea

remove load import

15ce005

danimtb removed their assignment Sep 20, 2018

danimtb mentioned this pull request Sep 21, 2018

Define Conan hooks management #3593

Closed

lasote approved these changes Sep 21, 2018

View reviewed changes

Merge branch 'develop' into feature/plugins_linters

70018ac

ghost assigned danimtb Sep 21, 2018

Merge branch 'develop' into feature/plugins_linters

9f249a6

memsharded reviewed Sep 21, 2018

View reviewed changes

removed assert

1172098

lasote merged commit c352051 into conan-io:develop Sep 24, 2018

ghost removed the stage: review label Sep 24, 2018

danimtb mentioned this pull request Sep 25, 2018

Plugins documentation conan-io/docs#846

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Conan plugins #3555

Conan plugins #3555

danimtb commented Sep 14, 2018 •

edited by lasote

jgsogo Sep 17, 2018

danimtb Sep 17, 2018

jgsogo Sep 19, 2018

lasote Sep 20, 2018

lasote Sep 20, 2018

lasote Sep 20, 2018

danimtb Sep 20, 2018

danimtb Sep 20, 2018

lasote Sep 20, 2018

memsharded commented Sep 21, 2018

lasote commented Sep 21, 2018

memsharded left a comment

memsharded Sep 21, 2018

danimtb commented Sep 24, 2018

lasote commented Sep 24, 2018

memsharded commented Sep 24, 2018

solvingj commented Oct 9, 2018

memsharded commented Oct 9, 2018

jgsogo commented Oct 9, 2018

Conan plugins #3555

Conan plugins #3555

Conversation

danimtb commented Sep 14, 2018 • edited by lasote

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

memsharded commented Sep 21, 2018

lasote commented Sep 21, 2018

memsharded left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danimtb commented Sep 24, 2018

lasote commented Sep 24, 2018

memsharded commented Sep 24, 2018

solvingj commented Oct 9, 2018

memsharded commented Oct 9, 2018

jgsogo commented Oct 9, 2018

danimtb commented Sep 14, 2018 •

edited by lasote