Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pipelines: not a readable file:bitbucket-pipelines.yml #1

Closed
pparchure opened this issue Aug 3, 2018 · 17 comments
Closed

pipelines: not a readable file:bitbucket-pipelines.yml #1

pparchure opened this issue Aug 3, 2018 · 17 comments

Comments

@pparchure
Copy link

Hello,
This is some great work!
But I am stuck at the first step, while running the YAML file. it says it's not a readable file. My YAML file uses a scala-sbt image with sbt commands in the script.
Could you please help me with this issue I am facing. Do I do some things differently?

Thanks,
Prathamesh

@ktomk
Copy link
Owner

ktomk commented Aug 3, 2018

What is the output of:

$ ls -al bitbucket-pipelines.yml

?

@ktomk ktomk added the feedback-appreciated curious for feedback and/or responses of open questions label Aug 3, 2018
@pparchure
Copy link
Author

I get
-rw-r--r--. 1 parchp01 cdsguest 461 Aug 1 12:50 bitbucket-pipelines.yml

@ktomk
Copy link
Owner

ktomk commented Aug 6, 2018

Hmm, that sounds looks interesting as it seems the file is readable.

Can you operate on it with other tools?

Can you read the file?

Can you change the file?

(all the same shell, in case you wonder)

/e: are you operating on a local file-system?

@pparchure
Copy link
Author

I can read the file using

php /home-path/bitbucket-pipelines.yml

but can't using the keyword 'pipelines'
I am running this on a linux based machine (local file system)

@ktomk
Copy link
Owner

ktomk commented Aug 6, 2018

A nice, and can you go into that directory and execute the following:

$ php -r 'var_dump(is_readable($argv[1]));' -- bitbucket-pipelines.yml

Also having the PHP version would be interesting:

$ php --version

And last but not least a bit of information about your system:

$ uname -a

@ktomk
Copy link
Owner

ktomk commented Aug 6, 2018

Ah an you could try the --debug switch with the pipelines utility. It might give a bit more info in your case, I'm not entirely sure though.

@pparchure
Copy link
Author

It still says not a readable file.
Has it anything to do with the docker image? is your php script is image specific? mine is sbt scala.

@pparchure
Copy link
Author

$ php -r 'var_dump(is_readable($argv[1]));' -- bitbucket-pipelines.yml --> bool(false)
$ php --version -->5.4.16
$ uname -a --> x86_64 x86_64 x86_64 GNU/Linux

This is what I get after executing those commands mentioned above.
Also, I read in your documentation that the image you have used i the default bitbucket-pipeline image. So would that affect reading my yaml file which has a scala-sbt docker image?

@ktomk
Copy link
Owner

ktomk commented Aug 6, 2018

Has it anything to do with the docker image?

No, the error comes before opening the file bitbucket-pipeline.yml for reading. At this point the pipelines utility does not even know which image is referenced inside the file.

is your php script is image specific?

No, it is not. It just uses any image you name there and will pass that exact name to docker.

Also, I read in your documentation that the image you have used i the default bitbucket-pipeline image. So would that affect reading my yaml file which has a scala-sbt docker image?

No, as written, the image name is not related to the error message. The error message (also confirmed by the "bool(false)") shows that the pipelines utility is seeing that bitbucket-pipeline.yml file as not readable. That is the only reason why you can't go on.

At this point I can only guess what is going on here. Something prevents showing the file as readable. As you say it is the local file-system and in the home-directory could it be that you have SE-Linux or similar running that is preventing showing readability of the file?

This is about your system, not the image. Just to make that explicit again.

Two more tests:

$ php -r 'var_dump(is_file($argv[1]));' -- bitbucket-pipelines.yml 
$ php -ddisplay_errors=1 -derror_reporting=-1 -r 'var_dump(strlen(file_get_contents($argv[1])));' -- bitbucket-pipelines.yml  

The first should give bool(true) and the second the number of bytes the file has. But please provide the output so I get more insights.

@pparchure
Copy link
Author

The first command gives : bool(true)
The second command gives: int(146)

Can you confirm the way I should run my .yml file?

@pparchure
Copy link
Author

Should the pipelines.phar file be inside my cloned project repo?

@ktomk
Copy link
Owner

ktomk commented Aug 6, 2018

Okay now I see it, the file has a SELinux security context. Most likely that one is preventing readability.

Check your system configuration and give the pipelines utility enough mojo in the SELinux security context to make is_readable return true.

This should solve your problem.

@ktomk
Copy link
Owner

ktomk commented Aug 6, 2018

You can try like this: RainLoop/rainloop-webmail#262 (comment)

ktomk added a commit that referenced this issue Aug 6, 2018
should make it working for files w/ acls / se linux contexts.

report: #1
ktomk added a commit that referenced this issue Aug 6, 2018
should make it working for files w/ acls / se linux contexts.

report: #1
ktomk added a commit that referenced this issue Aug 6, 2018
should make it working for files w/ acls / se linux contexts.

report: #1
@ktomk
Copy link
Owner

ktomk commented Aug 7, 2018

I'm preparing 0.0.18 which addresses this issue with falling back to a check if the file can be fopen()''ed correctly in case is_readable() fails.

@ktomk
Copy link
Owner

ktomk commented Aug 7, 2018

Version 0.0.18 is out and should be w/o that error on files with SE LInux context that make them appear non-readble by testing if the file can be read anyway. Please test it.

@ktomk
Copy link
Owner

ktomk commented Aug 8, 2018

@pparchure Any further feedback on this issue from your end?

@pparchure
Copy link
Author

@ktomk : sorry for the late response. It worked for me. Can now run the yaml file locally. That's some great work by you. Thank you. I'll stay in touch if I come across any problems in the future. Thanks a ton for your help.

@ktomk ktomk removed the feedback-appreciated curious for feedback and/or responses of open questions label Aug 13, 2018
@ktomk ktomk closed this as completed Aug 13, 2018
ktomk added a commit that referenced this issue Mar 7, 2020
workaround for error on docker pull:

    $ bin/pipelines --deploy mount --pipeline custom/lint-php
      +++ step #1
        name...........: (unnamed)
        effective-image: php:5.3
        container......: pipelines-1.no-name.custom-lint-php.pipelines
        container-id...: *failure*
    pipelines: setting up the container failed
    Unable to find image 'php:5.3' locally
    docker: Error response from daemon: unexpected EOF.
    See 'docker run --help'.
    exit status: 125
    The command "bin/pipelines --deploy mount --pipeline custom/lint-php" exited with 125.

it is only so far experiencing with the php:5.3, the problem is known for
CI, e.g. [1] or [2].

[1]: https://travis-ci.community/t/docker-pull-fails-often/7137
[2]: https://forums.docker.com/t/docker-pull-causes-error-response-from-daemon-unexpected-eof/88608
ktomk added a commit that referenced this issue Mar 7, 2020
workaround for error on docker pull:

    $ bin/pipelines --deploy mount --pipeline custom/lint-php
      +++ step #1
        name...........: (unnamed)
        effective-image: php:5.3
        container......: pipelines-1.no-name.custom-lint-php.pipelines
        container-id...: *failure*
    pipelines: setting up the container failed
    Unable to find image 'php:5.3' locally
    docker: Error response from daemon: unexpected EOF.
    See 'docker run --help'.
    exit status: 125
    The command "bin/pipelines --deploy mount --pipeline custom/lint-php" exited with 125.

it is only so far experiencing with the php:5.3, the problem is known for
CI, e.g. [1] or [2].

[1]: https://travis-ci.community/t/docker-pull-fails-often/7137
[2]: https://forums.docker.com/t/docker-pull-causes-error-response-from-daemon-unexpected-eof/88608
ktomk added a commit that referenced this issue Mar 7, 2020
workaround for error on docker pull:

    $ bin/pipelines --deploy mount --pipeline custom/lint-php
      +++ step #1
        name...........: (unnamed)
        effective-image: php:5.3
        container......: pipelines-1.no-name.custom-lint-php.pipelines
        container-id...: *failure*
    pipelines: setting up the container failed
    Unable to find image 'php:5.3' locally
    docker: Error response from daemon: unexpected EOF.
    See 'docker run --help'.
    exit status: 125
    The command "bin/pipelines --deploy mount --pipeline custom/lint-php" exited with 125.

it is only so far experiencing with the php:5.3, the problem is known for
CI, e.g. [1] or [2].

[1]: https://travis-ci.community/t/docker-pull-fails-often/7137
[2]: https://forums.docker.com/t/docker-pull-causes-error-response-from-daemon-unexpected-eof/88608
ktomk added a commit that referenced this issue Mar 7, 2020
workaround for error on docker pull:

    $ bin/pipelines --deploy mount --pipeline custom/lint-php
      +++ step #1
        name...........: (unnamed)
        effective-image: php:5.3
        container......: pipelines-1.no-name.custom-lint-php.pipelines
        container-id...: *failure*
    pipelines: setting up the container failed
    Unable to find image 'php:5.3' locally
    docker: Error response from daemon: unexpected EOF.
    See 'docker run --help'.
    exit status: 125
    The command "bin/pipelines --deploy mount --pipeline custom/lint-php" exited with 125.

it is only so far experiencing with the php:5.3, the problem is known for
CI, e.g. [1] or [2].

[1]: https://travis-ci.community/t/docker-pull-fails-often/7137
[2]: https://forums.docker.com/t/docker-pull-causes-error-response-from-daemon-unexpected-eof/88608
ktomk added a commit that referenced this issue Jun 1, 2020
the `--file` option accepts "-" (dash) as an argument to read the
pipelines yaml from standard input (stdin). a heredoc example:

    $ <<'PIPELINE' bin/pipelines --file -
    pipelines:
      default:
        - step:
            image: ktomk/pipelines:busybox
            script:
              - printenv | sort
    PIPELINE

    �+++ step #1

        name...........: (unnamed)
        effective-image: ktomk/pipelines:busybox
        container......: pipelines-1.no-name.default.pipelines
        container-id...: 3e0b182c1bcd

    �+++ copying files into container...


    �+ printenv | sort
    BITBUCKET_BUILD_NUMBER=0
    BITBUCKET_CLONE_DIR=/app
    BITBUCKET_COMMIT=0000000000000000000000000000000000000000
    BITBUCKET_REPO_OWNER=pipelines
    BITBUCKET_REPO_SLUG=pipelines
    ...

the `--file` option further accepts forms of process substitution now
as well:

    $ bin/pipelines --show  --file <(cat bitbucket-pipelines.yml) -v
    info: project directory is '/home/pipelines/workspace/pipelines'
    info: reading pipelines from /proc/self/fd/11
    PIPELINE ID                   STEP    IMAGE                             NAME
    default                       1       ktomk/pipelines:busybox           "pipeline features and introspection"
    custom/lint-php-5.3           1       php:5.3                           no-name
    custom/unit-tests-php-5.3     1       cespi/php-5.3:cli-latest          no-name
    ...

also test yaml parser implementations with a test-point for parser classes.
ktomk added a commit that referenced this issue Jun 2, 2020
the `--file` option accepts "-" (dash) as an argument to read the
pipelines yaml from standard input (stdin). a heredoc example:

    $ <<'PIPELINE' bin/pipelines --file -
    pipelines:
      default:
        - step:
            image: ktomk/pipelines:busybox
            script:
              - printenv | sort
    PIPELINE

    �+++ step #1

        name...........: (unnamed)
        effective-image: ktomk/pipelines:busybox
        container......: pipelines-1.no-name.default.pipelines
        container-id...: 3e0b182c1bcd

    �+++ copying files into container...


    �+ printenv | sort
    BITBUCKET_BUILD_NUMBER=0
    BITBUCKET_CLONE_DIR=/app
    BITBUCKET_COMMIT=0000000000000000000000000000000000000000
    BITBUCKET_REPO_OWNER=pipelines
    BITBUCKET_REPO_SLUG=pipelines
    ...

the `--file` option further accepts forms of process substitution now
as well:

    $ bin/pipelines --show  --file <(cat bitbucket-pipelines.yml) -v
    info: project directory is '/home/pipelines/workspace/pipelines'
    info: reading pipelines from /proc/self/fd/11
    PIPELINE ID                   STEP    IMAGE                             NAME
    default                       1       ktomk/pipelines:busybox           "pipeline features and introspection"
    custom/lint-php-5.3           1       php:5.3                           no-name
    custom/unit-tests-php-5.3     1       cespi/php-5.3:cli-latest          no-name
    ...

also test yaml parser implementations with a test-point for parser classes.
ktomk added a commit that referenced this issue Apr 15, 2022
when global docker option enabled, each step automatically gets docker
service.

missing so far as testing shows:


    $ bin/pipelines --file test/data/yml/options.yml --docker-client docker-42.42.1-binsh-test-stub
    +++ step #1

        name...........: "docker version"
        effective-image: ktomk/pipelines:busybox
        container......: pipelines-1.docker-version.default.pipelines
        container-id...: 415b9769dd1c

    +++ copying files into container...

    /bin/sh: docker: not found

    + docker --version
    script non-zero exit status: 127

needs fixing.
ktomk added a commit that referenced this issue Apr 15, 2022
when global docker option enabled, each step automatically gets docker
service.

missing so far as testing shows:


    $ bin/pipelines --file test/data/yml/options.yml --docker-client docker-42.42.1-binsh-test-stub
    +++ step #1

        name...........: "docker version"
        effective-image: ktomk/pipelines:busybox
        container......: pipelines-1.docker-version.default.pipelines
        container-id...: 415b9769dd1c

    +++ copying files into container...

    /bin/sh: docker: not found

    + docker --version
    script non-zero exit status: 127

needs fixing.
ktomk added a commit that referenced this issue Apr 15, 2022
when global docker option enabled, each step automatically gets docker
service.

missing so far as testing shows:


    $ bin/pipelines --file test/data/yml/options.yml --docker-client docker-42.42.1-binsh-test-stub
    +++ step #1

        name...........: "docker version"
        effective-image: ktomk/pipelines:busybox
        container......: pipelines-1.docker-version.default.pipelines
        container-id...: 415b9769dd1c

    +++ copying files into container...

    /bin/sh: docker: not found

    + docker --version
    script non-zero exit status: 127

needs fixing.
ktomk added a commit that referenced this issue Apr 16, 2022
when global docker option enabled, each step automatically gets docker
service.

missing so far as testing shows:


    $ bin/pipelines --file test/data/yml/options.yml --docker-client docker-42.42.1-binsh-test-stub
    +++ step #1

        name...........: "docker version"
        effective-image: ktomk/pipelines:busybox
        container......: pipelines-1.docker-version.default.pipelines
        container-id...: 415b9769dd1c

    +++ copying files into container...

    /bin/sh: docker: not found

    + docker --version
    script non-zero exit status: 127

needs fixing.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants