Skip to content
This repository has been archived by the owner on Jul 7, 2023. It is now read-only.

Travis log too long #523

Closed
martinpopel opened this issue Jan 17, 2018 · 6 comments
Closed

Travis log too long #523

martinpopel opened this issue Jan 17, 2018 · 6 comments
Labels

Comments

@martinpopel
Copy link
Contributor

Travis CI has 4MB limit on the log file, but current T2T tests exceed this limit.
As a result I cannot see the real cause why a given build failed.
Most of the log (if downloaded as a raw file) is full of download progress bars, e.g.

  Downloading mpmath-1.0.0.tar.gz (511kB)
�[?25l
�[K    2% |▋                               | 10kB 53.6MB/s eta 0:00:01
�[K    4% |█▎                              | 20kB 41.3MB/s eta 0:00:01
�[K    6% |██                              | 30kB 42.5MB/s eta 0:00:01
�[K    8% |██▋                             | 40kB 41.3MB/s eta 0:00:01
�[K    10% |███▏                            | 51kB 42.6MB/s eta 0:00:01

This is the first time I see this problem, but it seems that until a new pip version with --progress-bar off option is released, we should use a workaround pip install package | cat && exit ${PIPESTATUS[0]} or set -o pipefail; pip install package | cat.

martinpopel added a commit to martinpopel/tensor2tensor that referenced this issue Jan 17, 2018
@rsepassi
Copy link
Contributor

how about pip install -q to have the install be quiet?

@rsepassi rsepassi added the bug label Jan 18, 2018
@rsepassi
Copy link
Contributor

And git clone -q

@rsepassi
Copy link
Contributor

And thank you for spotting this.

@martinpopel
Copy link
Contributor Author

According to pypa/pip#2756 if we provide the --quiet option, pip does not display exceptions, which is a bit unfortunate for Travis. That said, it would be still better than the current state.
And yes, git clone -q is a good idea.

Unfortunately, it seems Travis is now completely broken. See e.g. https://travis-ci.org/tensorflow/tensor2tensor/builds/329575693 (note that a given commit just changes README.md).
In #524 I have tried to silence the pip, but Travis again failed for other reasons.

@rsepassi
Copy link
Contributor

rsepassi commented Jan 18, 2018 via email

martinpopel added a commit to martinpopel/tensor2tensor that referenced this issue Jan 18, 2018
@martinpopel
Copy link
Contributor Author

I changed | cat && exit ${PIPESTATUS[0]} into pip install -q in 61005b0
This commit (most probably disregarding its content) caused Travis rebuild and now it work and the raw log has just 82 KiB.
I was not able to find a place where to set git clone -q (git clone is done automatically by Travis).
So this issue can be closed once #524 is merged (sorry for combining two topics in one PR).

rsepassi pushed a commit that referenced this issue Jan 25, 2018
* no pip download progress bars in Travis log

see #523

* allow specifying --checkpoint_path with t2t-decoder

and allow keeping timestamp in that case.
This is needed for t2t-translate-all + t2t-bleu to work
as expected (I forgot to add this commit to #488).

* prevent tf.gfile.Glob crashes due to concurrent filesystem edits

tf.gfile.Glob may crash with
 tensorflow.python.framework.errors_impl.NotFoundError:
 xy/model.ckpt-1130761_temp_9cb4cb0b0f5f4382b5ea947aadfb7a40; No such file or directory
Let's use standard glob.glob instead, it seems to be more reliable.

* reintroducing FLAGS deleted by someone

this is needed for **locals() to work

* speedup BLEU tokenization

As I think about it, I would prefer my original implementation
https://github.com/tensorflow/tensor2tensor/blob/bb1173adce940e62c840970fa0f06f69fd9398db/tensor2tensor/utils/bleu_hook.py#L147-L157
But it seems there are some T2T/Google internal Python guidelines forbidding this,
so we have to live with the singleton.

* another solution of #523

* make save_checkpoints_secs work again

The functionality was broken during the adoption of TPU trainer_lib.py
instead of the original trainer_utils.py.
Currently, the default is to save checkpoints each 2000 steps,
while in previous T2T versions the default was each 10 minutes.

* adapt according to @rsepassi's review

* Update NotFoundError
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

2 participants