Copy_resources plugin not copying files/folders located outside of "dir_source_main_python" value #715

farooqind · 2020-04-27T16:33:30Z

@arcivanov My resources folder is not inside src/main/python but in src/main.
Plugin is not recognizing the path. It says it is copying but nothing gets copied in distribution.

Copying resources matching 'src/main/resources/**' from C:\work\xxx\xxxxx to c:\work\xxx\xxxx\target\dist\xxxxxxx-1.0

this is code in build.py:
@init
def initialize(project):
project.get_property("copy_resources_glob").append("src/main/resources/**")
project.set_property("copy_resources_target", "$dir_dist")

When I move the resources folder inside "src/main/python", it is copied properly to distribution.
@init
def initialize(project):
project.get_property("copy_resources_glob").append("src/main/python/resources/**")
project.set_property("copy_resources_target", "$dir_dist/resources")

arcivanov · 2020-04-27T19:29:55Z

I actually can't reproduce this at all...

[DEBUG] Registered Python environment 'system': CPython version 3.8.2.final.0 on linux in ['/home/arcivanov/.pyenv/versions/pyb-3.8.2/bin/python']
[DEBUG] Loading project module from /tmp/IntegrationTestSupportlsctx1nf/build.py
[DEBUG] Setting up plugins VEnv at '/tmp/IntegrationTestSupportlsctx1nf/.pybuilder/plugins/cpython-3.8.2.final.0'
[DEBUG] Registered Python environment 'pybuilder': CPython version 3.8.2.final.0 on linux in ['/tmp/IntegrationTestSupportlsctx1nf/.pybuilder/plugins/cpython-3.8.2.final.0/bin/python']
[DEBUG] Patched __import__ system to defer plugin loading
[DEBUG] Loading plugin 'python.core'
[DEBUG] Trying to load direct plugin 'python.core', module 'pybuilder.plugins.python.core_plugin'
[DEBUG] Found direct plugin 'python.core', module 'pybuilder.plugins.python.core_plugin'
[DEBUG] Loading plugin 'copy_resources'
[DEBUG] Trying to load direct plugin 'copy_resources', module 'pybuilder.plugins.copy_resources_plugin'
[DEBUG] Found direct plugin 'copy_resources', module 'pybuilder.plugins.copy_resources_plugin'
[DEBUG] Loading plugin 'core'
[DEBUG] Trying to load direct plugin 'core', module 'pybuilder.plugins.core_plugin'
[DEBUG] Found direct plugin 'core', module 'pybuilder.plugins.core_plugin'
[DEBUG] Found task 'clean' with dependencies []
[DEBUG] Registering task 'clean'
[DEBUG] Found task 'compile_sources' with dependencies [prepare]
[DEBUG] Registering task 'compile_sources'
[DEBUG] Registering initializer 'init'
[DEBUG] Found task 'install' with dependencies [package, publish(optional)]
[DEBUG] Registering task 'install'
[DEBUG] Found task 'package' with dependencies [compile_sources, run_unit_tests(optional)]
[DEBUG] Registering task 'package'
[DEBUG] Found task 'prepare' with dependencies []
[DEBUG] Registering task 'prepare'
[DEBUG] Found task 'print_module_path' with dependencies []
[DEBUG] Registering task 'print_module_path'
[DEBUG] Found task 'print_scripts_path' with dependencies []
[DEBUG] Registering task 'print_scripts_path'
[DEBUG] Found task 'publish' with dependencies [package, verify(optional)]
[DEBUG] Registering task 'publish'
[DEBUG] Found task 'run_integration_tests' with dependencies [package]
[DEBUG] Registering task 'run_integration_tests'
[DEBUG] Found task 'run_unit_tests' with dependencies [compile_sources]
[DEBUG] Registering task 'run_unit_tests'
[DEBUG] Found task 'verify' with dependencies [run_integration_tests(optional)]
[DEBUG] Registering task 'verify'
[DEBUG] Found task 'prepare' with dependencies []
[DEBUG] Registering task 'prepare'
[DEBUG] Registering initializer 'init_python_directories'
[DEBUG] Found task 'package' with dependencies []
[DEBUG] Registering task 'package'
[DEBUG] Registering initializer 'init_copy_resources_plugin'
[DEBUG] Found task 'package' with dependencies []
[DEBUG] Registering task 'package'
[DEBUG] Have loaded plugins python.core, copy_resources, core
[DEBUG] Registering initializer 'init'
[DEBUG] Adding 'prepare' as a dependency of task 'compile_sources'
[DEBUG] Adding 'package' as a dependency of task 'install'
[DEBUG] Adding 'publish(optional)' as a dependency of task 'install'
[DEBUG] Adding 'compile_sources' as a dependency of task 'package'
[DEBUG] Adding 'run_unit_tests(optional)' as a dependency of task 'package'
[DEBUG] Adding 'package' as a dependency of task 'publish'
[DEBUG] Adding 'verify(optional)' as a dependency of task 'publish'
[DEBUG] Adding 'package' as a dependency of task 'run_integration_tests'
[DEBUG] Adding 'compile_sources' as a dependency of task 'run_unit_tests'
[DEBUG] Adding 'run_integration_tests(optional)' as a dependency of task 'verify'
[DEBUG] Executing initializer 'init' from 'pybuilder.plugins.core_plugin'
[DEBUG] Executing initializer 'init_python_directories' from 'pybuilder.plugins.python.core_plugin'
[DEBUG] Executing initializer 'init_copy_resources_plugin' from 'pybuilder.plugins.copy_resources_plugin'
[DEBUG] Executing initializer 'init' from 'build'
[DEBUG] Project properties: 
                                 basedir : /tmp/IntegrationTestSupportlsctx1nf
                     copy_resources_glob : ['src/main/resources/**']
                   copy_resources_target : $dir_dist
                                dir_dist : $dir_target/dist/IntegrationTestSupportlsctx1nf-1.0.dev0
                        dir_dist_scripts : scripts
                        dir_install_logs : $dir_logs/install_dependencies
                                dir_logs : $dir_target/logs
                             dir_reports : $dir_target/reports
                  dir_source_main_python : src/main/python
                 dir_source_main_scripts : src/main/scripts
                              dir_target : target
        install_dependencies_constraints : constraints_file
    install_dependencies_extra_index_url : None
          install_dependencies_index_url : None
install_dependencies_insecure_installation : []
       install_dependencies_trusted_host : None
            install_dependencies_upgrade : False
                             pip_verbose : 0
                           refresh_venvs : False
                            remote_debug : 0
                              venv_clean : False
                       venv_dependencies : {}
                              venv_names : ['build', 'test']
                                 verbose : False
[DEBUG] Execution plan is prepare, compile_sources, run_unit_tests, package
[INFO]  Building IntegrationTestSupportlsctx1nf version 1.0.dev0
[INFO]  Executing build in /tmp/IntegrationTestSupportlsctx1nf
[INFO]  Going to execute task package
[DEBUG] Executing task 'prepare'
[DEBUG] Executing subtask of 'prepare' from 'pybuilder.plugins.core_plugin'
[DEBUG] Creating target directory /tmp/IntegrationTestSupportlsctx1nf/target
[DEBUG] Creating reports directory /tmp/IntegrationTestSupportlsctx1nf/target/reports
[DEBUG] Executing subtask of 'prepare' from 'pybuilder.plugins.python.core_plugin'
[DEBUG] Creating log directory '/tmp/IntegrationTestSupportlsctx1nf/target/logs/install_dependencies'
[INFO]  Creating target 'build' VEnv in '/tmp/IntegrationTestSupportlsctx1nf/target/venv/build/cpython-3.8.2.final.0'
[DEBUG] Registered Python environment 'build': CPython version 3.8.2.final.0 on linux in ['/tmp/IntegrationTestSupportlsctx1nf/target/venv/build/cpython-3.8.2.final.0/bin/python']
[INFO]  Creating target 'test' VEnv in '/tmp/IntegrationTestSupportlsctx1nf/target/venv/test/cpython-3.8.2.final.0'
[DEBUG] Registered Python environment 'test': CPython version 3.8.2.final.0 on linux in ['/tmp/IntegrationTestSupportlsctx1nf/target/venv/test/cpython-3.8.2.final.0/bin/python']
[DEBUG] Executing task 'compile_sources'
[DEBUG] Executing subtask of 'compile_sources' from 'pybuilder.plugins.core_plugin'
[DEBUG] Executing task 'run_unit_tests'
[DEBUG] Executing subtask of 'run_unit_tests' from 'pybuilder.plugins.core_plugin'
[DEBUG] Executing task 'package'
[DEBUG] Executing subtask of 'package' from 'pybuilder.plugins.core_plugin'
[DEBUG] Executing subtask of 'package' from 'pybuilder.plugins.python.core_plugin'
[DEBUG] Creating directory /tmp/IntegrationTestSupportlsctx1nf/target/dist/IntegrationTestSupportlsctx1nf-1.0.dev0
[INFO]  Building distribution in /tmp/IntegrationTestSupportlsctx1nf/target/dist/IntegrationTestSupportlsctx1nf-1.0.dev0
[INFO]  Copying scripts to /tmp/IntegrationTestSupportlsctx1nf/target/dist/IntegrationTestSupportlsctx1nf-1.0.dev0/scripts
[DEBUG] Executing subtask of 'package' from 'pybuilder.plugins.copy_resources_plugin'
[INFO]  Copying resources matching 'src/main/resources/**' from /tmp/IntegrationTestSupportlsctx1nf to /tmp/IntegrationTestSupportlsctx1nf/target/dist/IntegrationTestSupportlsctx1nf-1.0.dev0
[DEBUG] Copying resource src/main/resources/eggs
[DEBUG] Copying resource src/main/resources/spam
[DEBUG] Copying resource src/main/resources/foo/bar

arcivanov · 2020-04-27T19:31:10Z

from pybuilder.core import use_plugin, init

use_plugin("python.core")
use_plugin("copy_resources")

@init
def init (project):
    project.get_property("copy_resources_glob").append("src/main/resources/**")
    project.set_property("copy_resources_target", "$dir_dist")

farooqind · 2020-04-27T20:08:17Z

@arcivanov I tried again..here are details..it still doesn't work. Distribution is not having resources folder.

farooqind · 2020-04-27T20:17:20Z

@arcivanov However this works fine. ie providing the file directly at root, it copies that file.

arcivanov · 2020-04-27T22:21:46Z

Confirmed on Windows: https://travis-ci.org/github/pybuilder/pybuilder/jobs/680308181

farooqind · 2020-04-27T23:48:43Z

For me it doesn't work on Linux as well.
I am using Jenkins for the build. First I am creating virtual environment and then running the build on Linux server. As usual it says it is copying but nothing appears in the tar file..

[INFO] Copying resources matching 'src/main/resources/**' from /disk1/jenkins/XXXXXX/workspace/ABC/abc_test to /disk1/jenkins/XXXXXX/workspace/ABC/abc_test/target/dist/abc_test-1.0

[INFO] Copying resources matching 'src/main/resources/**' from /disk1/jenkins/XXXXX/workspace/ABC/abc_test to /disk1/jenkins/XXXXX/workspace/ABC/abc_test/target/dist/abc_test-1.0
[DEBUG] Copying resource src/main/resources/.gitignore
[DEBUG] Copying resource src/main/resources/env.json
[DEBUG] Copying resource src/main/resources/aa/a.json
[DEBUG] Copying resource src/main/resources/bb/b.json
[DEBUG] Copying resource src/main/resources/cc/c.json
[DEBUG] Copying resource src/main/resources/cc/yy/d.json
[DEBUG] Copying resource src/main/resources/cc/zz/e.json
[DEBUG] Copying resource src/main/resources/cc/bb/f.json
[DEBUG] Copying resource src/main/resources/cc/kk/g.json

arcivanov · 2020-04-28T00:06:51Z

@farooqind "Not copying" and "Not appearing in tar file" are DIFFERENT things. I have specifically asked you here where the files did not appear.

On Linux copying works verifiably. On Windows it appears not to, I'm currently investigating why.

farooqind · 2020-04-28T00:40:35Z

@arcivanov Ultimate aim is to get resources folder as part of tar file. As per my understanding & observation, whatever appears in distribution folder, is also appearing in tar file for distribution. Please correct me if my understanding is wrong.

So I checked on Jenkins Linux server, it is copying the files. However, it is not putting those folders/files into final .tar.gz file. Apologies for the confusion.
So there are 2 points:

recursive copying not working on windows
copied files not being put into .tar.gz file

Replace custom RE-based glob with real `glob.iglob` to make behavior standard. Fix itest harness moronic path manipulation. fixes pybuilder#715

Run tests to see whether there is a Windows issue for #715

farooqind mentioned this issue Apr 27, 2020

question: how should I handle additional resources? #663

Open

arcivanov added a commit to arcivanov/pybuilder that referenced this issue Apr 27, 2020

Run tests to see whether there is a Windows issue for pybuilder#715

a3df9a4

arcivanov added a commit to arcivanov/pybuilder that referenced this issue Apr 27, 2020

Run tests to see whether there is a Windows issue for pybuilder#715

b1298f3

arcivanov added a commit to arcivanov/pybuilder that referenced this issue Apr 27, 2020

Run tests to see whether there is a Windows issue for pybuilder#715

0492004

arcivanov added bug windows specific labels Apr 27, 2020

arcivanov added a commit to arcivanov/pybuilder that referenced this issue Apr 28, 2020

Run tests to see whether there is a Windows issue for pybuilder#715

1babff3

arcivanov closed this as completed in 5631d89 Apr 28, 2020

arcivanov added a commit that referenced this issue Apr 28, 2020

Merge pull request #716 from arcivanov/issue_715

b5c7713

Run tests to see whether there is a Windows issue for #715

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Copy_resources plugin not copying files/folders located outside of "dir_source_main_python" value #715

Copy_resources plugin not copying files/folders located outside of "dir_source_main_python" value #715

farooqind commented Apr 27, 2020 •

edited

Loading

arcivanov commented Apr 27, 2020

arcivanov commented Apr 27, 2020

farooqind commented Apr 27, 2020 •

edited

Loading

farooqind commented Apr 27, 2020

arcivanov commented Apr 27, 2020

farooqind commented Apr 27, 2020 •

edited

Loading

arcivanov commented Apr 28, 2020 •

edited

Loading

farooqind commented Apr 28, 2020 •

edited

Loading

Copy_resources plugin not copying files/folders located outside of "dir_source_main_python" value #715

Copy_resources plugin not copying files/folders located outside of "dir_source_main_python" value #715

Comments

farooqind commented Apr 27, 2020 • edited Loading

arcivanov commented Apr 27, 2020

arcivanov commented Apr 27, 2020

farooqind commented Apr 27, 2020 • edited Loading

farooqind commented Apr 27, 2020

arcivanov commented Apr 27, 2020

farooqind commented Apr 27, 2020 • edited Loading

arcivanov commented Apr 28, 2020 • edited Loading

farooqind commented Apr 28, 2020 • edited Loading

farooqind commented Apr 27, 2020 •

edited

Loading

farooqind commented Apr 27, 2020 •

edited

Loading

farooqind commented Apr 27, 2020 •

edited

Loading

arcivanov commented Apr 28, 2020 •

edited

Loading

farooqind commented Apr 28, 2020 •

edited

Loading