Run tests on Travis CI #224

abitrolly · 2020-07-29T17:05:17Z

Travis status from my fork https://travis-ci.com/github/abitrolly/cheat.sh/builds

They don't really work well.

Because `docker-compose` is not capable of checking open ports docker/compose#374

abitrolly · 2020-07-29T17:58:15Z

@chubin now I am not sure what's going on. More output wouldn't hurt. Like the number of test and the command what is being executed.

https://travis-ci.com/github/abitrolly/cheat.sh/builds/177681447#L425

abitrolly · 2020-07-29T18:31:38Z

It will take a lot of time for me to understand the shell runner. For example, what does this line do?

cheat.sh/tests/run-tests.sh

Line 36 in c17183c

cat -n tests.txt | sed -n "$(echo "$*" | sed 's/ /p; /g;s/$/p/')"

chubin · 2020-07-29T18:45:19Z

@abitrolly I believe you are doing it almost right, but it tries to use the standalone mode instead of the curl mode.

Let us take a look at the first test output:

CHEATSH_CACHE_TYPE=none python ../lib/standalone.py python/:list

Ok, it uses the standalone mode here, and we should curl instead.
If you set:

CHEATSH_TEST_STANDALONE=NO

What does this mean:

1,18d0
< 1line
< :learn
< :list
< Advanced
< Classes
< Comments
< Control_Flow
< Functions
< Modules
< Primitive_Datatypes_and_Operators
< Variables_and_Collections
< doc
< func
< hello
< lambda
< loops
< recursion
< rosetta/

?
This is a diff between what you've wanted and what you've got. Because it is running in a standalone mode
which delivers nothing, it is basically the whole expected page.

Regarding this:

cat -n tests.txt | sed -n "$(echo "$*" | sed 's/ /p; /g;s/$/p/')"`

The line looks so scary but it is pretty trivial:

 s{ }{p; }g;
 s{$}{p}

what yields:

1 2 3 => 1p; 2p; 3p;

which then is used as a short script for the outer sed printing the selected tests.
I had to document this better; here we just extract the tests that were passed as an arguments
to the test.sh.

So, to summarize, CHEATSH_TEST_STANDALONE=NO should be sufficient

abitrolly · 2020-07-29T19:01:23Z

@chubin what is the standalone mode? Does it mean ignoring the server?

chubin · 2020-07-29T19:03:54Z

Yes, in the standalone mode you can use it without the server (say during a flight or in a submarine);
currently it does not support programming language queries in this mode though, only cheat sheet queries.
See #14 for some details

abitrolly · 2020-07-30T13:10:07Z

I've discovered the Travis has numerous issues with ANSI escaping. Basically it is doing the same transformation of ANSI sequences as cheat.sh does to convert ANSI colored logs to HTML, and it looks like a not very maintainable hack https://github.com/travis-ci/travis-web/blob/master/app/utils/log.js#L236

abitrolly · 2020-07-30T18:06:16Z

Filled the bug https://travis-ci.community/t/ansi-logs-missing-colors-and-formatting-problems/9489

SuperSandro2000 · 2020-07-30T18:10:38Z

.travis.yml

@@ -7,3 +7,9 @@ before_install:
  - docker-compose up -d
  - docker ps
  - docker images
+
+script:
+  - sleep 3


sleep is probably not a good way to know when the server is up. If you run it locally on a slower machine this can easily fail.

Thanks, cap. So any ideas how to fix that - https://travis-ci.com/github/abitrolly/cheat.sh/builds/177678164#L390? I checked - it is non-trivial

Retry after one second up to X if the exit code is 52 maybe?

How exit code 52 is related?

The command "curl http://localhost:8002" exited with 52.

Then maybe better check until a port is accepting connections?

tests/run-tests.sh

abitrolly · 2020-07-30T18:49:23Z

@SuperSandro2000 thanks for the comments on things not catched by shellcheck. It always good to have more pair of eyes. Any ideas why the tests fail?

And remove temporary `curl cht.sh` call

chubin/cheat.sheets@d7473ac#diff-8d0f560b012648accbca6af7a3356d61

This would allow later inspection of stdout. Assigned names to image and container to avoid confusion. Build is done separately to clean up the output.

Run Python 3 in unbuffered mode to see lines in `docker logs` as soon as they appear. When Python 3 is not attached to a terminal, it turns on the buffering, like when `docker` runs in a background.

Default `localhost:6379` doesn't work there. Trying to set REDIS_HOST advertised in `run-tests.sh`

abitrolly · 2020-07-31T16:31:26Z

redis.exceptions.ConnectionError: Error 99 connecting to localhost:6379. Address not available.

2020-07-31T14:00:00Z {'REMOTE_ADDR': '172.18.0.1', 'REMOTE_PORT': '42306', 'HTTP_HOST': 'localhost:8002', (hidden keys: 20)} failed with ConnectionError

At last! Some progress.

abitrolly · 2020-07-31T16:56:32Z

REDIS_HOST env setting has no effect. Reading up on host cheat.sh config works https://github.com/chubin/cheat.sh/blob/f5d5fae71218e771e84b896600b2504c0d8814ec/lib/config.py

https://github.com/chubin/cheat.sh/blob/f5d5fae71218e771e84b896600b2504c0d8814ec/lib/config.py

The proper config name for Redis host is CHEATSH_CACHE_REDIS_HOST and for disabling cache the var is CHEATSH_CACHE_TYPE.

abitrolly · 2020-07-31T23:13:55Z

@chubin thsi is it. Travis runs test properly. The rest - updating and fixing the tests - could be done in subsequent PRs.

abitrolly · 2020-07-31T23:14:06Z

https://travis-ci.com/github/abitrolly/cheat.sh/builds/178022132

https://github.com/andymccurdy/redis-py#client-classes-redis-and-strictredis

SuperSandro2000 · 2020-08-01T07:20:45Z

tests/run-tests.sh

@@ -59,19 +71,24 @@ while read -r number test_line; do

  if [ "$test_standalone" = YES ]; then
    test_line="${test_line//cht.sh /}"
-    "${PYTHON}" ../lib/standalone.py "$test_line" > "$TMP" 2> /dev/null
+    [[ $show_details == YES ]] && echo "${PYTHON} ../lib/standalone.py $test_line"


You could convert this to a function which takes the string as an argument.

I am not sure I am that bash proficient. Show an example?

log() { [[ $show_details == YES ]] && echo "$1" } log("${PYTHON} ../lib/standalone.py $test_line")

chubin · 2020-08-06T06:20:40Z

@abitrolly

The PR is merged, thank you for the great work!

I've fixed the tests, they are almost all green now, except two tests, and they have, probably, the same problem.
Strange enough, this problem occurs only in the test environment, but not when running interactively.

That is how the correct expected behavior looks like:

$ curl -s http://localhost:8002/python/copy+file?QT
from shutil import copyfile

copyfile(src, dst)

Q means quiet mode, and T means no terminal,
but in the test environment the options seems to be ignored.

Could you please take a look at it?
If you will not find anything, then we will investigate deeper

Last build output:

https://travis-ci.org/github/chubin/cheat.sh/builds/715398758

abitrolly · 2020-08-06T07:43:04Z

@chubin I will try to quote them. Maybe there are some problems with shell expansion.

chubin · 2020-08-06T07:54:06Z

@abitrolly I don't think that this helps; try more debug output better instead

abitrolly · 2020-08-06T07:58:33Z

@chubin the part I don't understand is the usage of eval - could it be that eval does something to ? like expanding it as a shell pattern?

cheat.sh/tests/run-tests.sh

Line 85 in 9f99bec

eval "curl -s $CHTSH_URL/$test_line" > "$TMP"

abitrolly · 2020-08-06T11:04:53Z

@chubin on my local instance accessing http://localhost:8002/python/copy+file complains that I am offline. Strange that it shows 200 status code. I haven't found a way to debug Flask yet. logging doesn't output anything on screen.

import logging
logging.error(url)
logging.error(response)

chubin · 2020-10-12T15:51:40Z

@abitrolly I am back online; sorry for the disappearance.
I will try to take a look at it and fix the problem

abitrolly · 2020-10-13T05:10:11Z

@chubin no pressure. I will be offline for at least two weeks too. )

abitrolly added 4 commits July 29, 2020 19:56

Run tests on Travis CI

ce7b4c8

Test server is up with curl before running tests

40d18da

cd to tests/ dir when runner is executed with relpath

06d8fa7

Wait 3 seconds to let server to start

3905047

Because `docker-compose` is not capable of checking open ports docker/compose#374

abitrolly added 5 commits July 29, 2020 22:44

Use python3 command if virtualenv doesn't exist

c515c56

Detect Python version using Python

1a1624e

Turn off standalone mode to run tests against remote server

1a5aa64

Show which test is being run

60ae85e

Add color to printed test names

09dff20

abitrolly mentioned this pull request Jul 30, 2020

Update Travis config #223

Merged

Run curl https://cht.sh` to pinpoint Travis problems with ANSI

c0bbc24

SuperSandro2000 reviewed Jul 30, 2020

View reviewed changes

abitrolly added 10 commits July 30, 2020 22:20

Travis doesn't support bright colors, use 8 colors palette

4610690

And remove temporary `curl cht.sh` call

Show test command executed when $show_details == YES

1387f8b

Use unified diff format

5737a16

Use http://localhost:8002 for default tests CHTSH_URL

a0d344c

Pipe colored diffs

e0f84e8

Fix btrfs test drift as trailing whitespaces were fixed

1032623

chubin/cheat.sheets@d7473ac#diff-8d0f560b012648accbca6af7a3356d61

Run cheat.sh service (app) by docker-compose in debug mode

be9d3a7

This would allow later inspection of stdout. Assigned names to image and container to avoid confusion. Build is done separately to clean up the output.

docker-compose run requires explicit port permission

6be6a25

Try to show container log in case of error on Travis

9a495a7

Try --no-color to avoid log output being stripped by Travis

5dab632

abitrolly added 7 commits July 31, 2020 13:31

Modern DevOps is a PITA

10bfaab

Run Python 3 in unbuffered mode to see lines in `docker logs` as soon as they appear. When Python 3 is not attached to a terminal, it turns on the buffering, like when `docker` runs in a background.

Try docker-compose logs with no color

2e1d804

View container log without docker-compose

f62fb51

Check docker logs with unbuffered Python again

ca55415

Add --debug option to chtsh container the docker-compose way

198a2c4

Override files needs a version too

acd07d2

Now fixing connection to Redis on Travis Docker

ee273e2

Default `localhost:6379` doesn't work there. Trying to set REDIS_HOST advertised in `run-tests.sh`

abitrolly added 2 commits August 1, 2020 00:41

Set Redis host using cheat.sh config framework

91320d9

https://github.com/chubin/cheat.sh/blob/f5d5fae71218e771e84b896600b2504c0d8814ec/lib/config.py

Set proper config names and update docker status in README.md

36a7cd3

The proper config name for Redis host is CHEATSH_CACHE_REDIS_HOST and for disabling cache the var is CHEATSH_CACHE_TYPE.

abitrolly added 2 commits August 1, 2020 06:06

run-tests.sh Quotes are not needed in default param

ca88810

Redis doesn't need to be Strict

b74707a

https://github.com/andymccurdy/redis-py#client-classes-redis-and-strictredis

SuperSandro2000 reviewed Aug 1, 2020

View reviewed changes

chubin added 2 commits August 6, 2020 06:27

Merge branch 'master' into run-tests

18e851c

minifix

7578ab7

chubin merged commit cd905f7 into chubin:master Aug 6, 2020

abitrolly deleted the run-tests branch August 6, 2020 07:43

chubin mentioned this pull request Oct 12, 2020

Posix sh #237

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run tests on Travis CI #224

Run tests on Travis CI #224

abitrolly commented Jul 29, 2020 •

edited

abitrolly commented Jul 29, 2020

abitrolly commented Jul 29, 2020 •

edited

chubin commented Jul 29, 2020

abitrolly commented Jul 29, 2020

chubin commented Jul 29, 2020

abitrolly commented Jul 30, 2020

abitrolly commented Jul 30, 2020 •

edited

SuperSandro2000 Jul 30, 2020

abitrolly Jul 30, 2020

SuperSandro2000 Jul 30, 2020

abitrolly Jul 31, 2020

SuperSandro2000 Jul 31, 2020

abitrolly Jul 31, 2020

abitrolly commented Jul 30, 2020

abitrolly commented Jul 31, 2020

abitrolly commented Jul 31, 2020

abitrolly commented Jul 31, 2020

abitrolly commented Jul 31, 2020

SuperSandro2000 Aug 1, 2020

abitrolly Aug 1, 2020

SuperSandro2000 Aug 1, 2020

chubin commented Aug 6, 2020

abitrolly commented Aug 6, 2020

chubin commented Aug 6, 2020

abitrolly commented Aug 6, 2020

abitrolly commented Aug 6, 2020

chubin commented Oct 12, 2020

abitrolly commented Oct 13, 2020

Run tests on Travis CI #224

Run tests on Travis CI #224

Conversation

abitrolly commented Jul 29, 2020 • edited

abitrolly commented Jul 29, 2020

abitrolly commented Jul 29, 2020 • edited

chubin commented Jul 29, 2020

abitrolly commented Jul 29, 2020

chubin commented Jul 29, 2020

abitrolly commented Jul 30, 2020

abitrolly commented Jul 30, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abitrolly commented Jul 30, 2020

abitrolly commented Jul 31, 2020

abitrolly commented Jul 31, 2020

abitrolly commented Jul 31, 2020

abitrolly commented Jul 31, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

chubin commented Aug 6, 2020

abitrolly commented Aug 6, 2020

chubin commented Aug 6, 2020

abitrolly commented Aug 6, 2020

abitrolly commented Aug 6, 2020

chubin commented Oct 12, 2020

abitrolly commented Oct 13, 2020

abitrolly commented Jul 29, 2020 •

edited

abitrolly commented Jul 29, 2020 •

edited

abitrolly commented Jul 30, 2020 •

edited