Feature Request: Relax the length of text to be logged

[stonelazy]

Is your feature request related to a problem? Please describe.

My feature request is related to a problem, Neptune ignores any text that is longer than 1000 characters to be logged.
Usually when the training crashes we have a long list of stack traces and that can easily be more than 1000 characters, atleast for these categories neptune should relax the restriction.

Epoch 0: : 567it [01:14,  7.59it/s, loss=0.543, v_num=D-80]^CError executing job with overrides: ['data.train_metadata_file=[/data/kp/vad/vad_dvc_data/voa/training/train.csv,/data/kp/vad/pp/dns_noise/training/train.csv]
Execution of main took, 583.5295114517212
Failed to send channel value.
Traceback (most recent call last):
  File "/home/kp/anaconda3/envs/gamd5-kp2/lib/python3.8/site-packages/neptune/internal/channels/channels_values_sender.py", line 157, in _send_values
    self._experiment._send_channels_values(channels_with_values)
  File "/home/kp/anaconda3/envs/gamd5-kp2/lib/python3.8/site-packages/neptune/experiments.py", line 1032, in _send_channels_values
    self._backend.send_channels_values(self, channels_with_values)
  File "/home/kp/anaconda3/envs/gamd5-kp2/lib/python3.8/site-packages/neptune/utils.py", line 271, in wrapper
    return func(*args, **kwargs)
  File "/home/kp/anaconda3/envs/gamd5-kp2/lib/python3.8/site-packages/neptune/internal/api_clients/hosted_api_clients/hosted_alpha_leaderboard_api_client.py", line 377, in send_channels_values
    self._execute_operations(experiment, send_operations)
  File "/home/kp/anaconda3/envs/gamd5-kp2/lib/python3.8/site-packages/neptune/utils.py", line 271, in wrapper
    return func(*args, **kwargs)
  File "/home/kp/anaconda3/envs/gamd5-kp2/lib/python3.8/site-packages/neptune/internal/api_clients/hosted_api_clients/hosted_alpha_leaderboard_api_client.py", line 638, in _execute_operations
    raise ExperimentOperationErrors(errors=errors)
neptune.api_exceptions.ExperimentOperationErrors: Caused by:
	* Invalid point for string series: monitoring/stderr : Text longer than 1000 characters was truncated
Traceback (most recent call last):
  File "train.py", line 29, in <module>
    main()  # pylint: disable=no-value-for-parameter
  File "/home/kp/Remote/zspeech/zspeech/utils/date_time_utils.py", line 36, in wrapper
    function()
  File "/home/kp/anaconda3/envs/gamd5-kp2/lib/python3.8/site-packages/hydra/main.py", line 49, in decorated_main
    _run_hydra(
  File "/home/kp/anaconda3/envs/gamd5-kp2/lib/python3.8/site-packages/hydra/_internal/utils.py", line 367, in _run_hydra
    run_and_report(
  File "/home/kp/anaconda3/envs/gamd5-kp2/lib/python3.8/site-packages/hydra/_internal/utils.py", line 214, in run_and_report
    raise ex
  File "/home/kp/anaconda3/envs/gamd5-kp2/lib/python3.8/site-packages/hydra/_internal/utils.py", line 211, in run_and_report
    return func()
  File "/home/kp/anaconda3/envs/gamd5-kp2/lib/python3.8/site-packages/hydra/_internal/utils.py", line 368, in <lambda>
    lambda: hydra.run(
  File "/home/kp/anaconda3/envs/gamd5-kp2/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 110, in run
    _ = ret.return_value
  File "/home/kp/anaconda3/envs/gamd5-kp2/lib/python3.8/site-packages/hydra/core/utils.py", line 233, in return_value
    raise self._return_value
  File "/home/kp/anaconda3/envs/gamd5-kp2/lib/python3.8/site-packages/hydra/core/utils.py", line 160, in run_job
    ret.return_value = task_function(task_cfg)
  File "train.py", line 25, in main
    Trainer.train_from_cfg(cfg)
  File "/home/kp/Remote/zspeech/zspeech/training.py", line 384, in train_from_cfg
    raise ex
  File "/home/kp/Remote/zspeech/zspeech/training.py", line 382, in train_from_cfg
    trainer.train(experiment, data)
  File "/home/kp/Remote/zspeech/zspeech/training.py", line 276, in train
    self.trainer.test()
  File "/home/kp/anaconda3/envs/gamd5-kp2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 703, in test
    self.tested_ckpt_path = self.__load_ckpt_weights(ckpt_path)
  File "/home/kp/anaconda3/envs/gamd5-kp2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1143, in __load_ckpt_weights
    raise MisconfigurationException(
pytorch_lightning.utilities.exceptions.MisconfigurationException: `.test(ckpt_path="best")` is set but `ModelCheckpoint` is not configured to save the best model.

[Herudaio]

Hi @stonelazy !

In case of an exception stack trace is logged as a separate field in the monitoring namespace. The stderr will be trimmed to the limit, but the full stack trace is still captured. Could you check if it was captured correctly for this run?

[stonelazy]

Am sorry, i have trashed the experiment and so i couldn't confirm the behavior. However, allow me close the issue and reopen if am not able to view the trace in Neptune.

[Ulipenitz]

I am also interested in this feature!

As I am uploading a dataset (which does not fit on my local disk) to the project, I am uploading the dataset in a loop to the project like this:

project[DATA_PATH].append(
                stringify_unsupported(
                    {
                        "tokens": ["text",...., "text"],
                        "ner_tags": ["tag",...,"tag"]
                    }
                )
            )

Truncation to 1000 characters destroys my dataset.
As of my knowledge, there is no other way to upload a dataset from memory (without saving to a local file) directly, so this feature would be great!