Skip to content
This repository has been archived by the owner on Jun 6, 2023. It is now read-only.

enable non-verbose logging of relaxation (e.g. Relax(verbose=0)) #11

Closed
sgbaird opened this issue Jun 17, 2022 · 7 comments
Closed

enable non-verbose logging of relaxation (e.g. Relax(verbose=0)) #11

sgbaird opened this issue Jun 17, 2022 · 7 comments

Comments

@sgbaird
Copy link
Contributor

sgbaird commented Jun 17, 2022

This isn't time-sensitive (I'm grateful it's running!), but I'm planning to loop through many structures, and the default output I get for a single relaxation is:

2022-06-16 19:38:02.863217: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found
2022-06-16 19:38:02.866924: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.       
Skipping registering GPU devices...
2022-06-16 19:38:02.875852: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("PartitionedCall:4", shape=(None,), dtype=int32), values=Tensor("PartitionedCall:3", shape=(None, 3, 3), dtype=float32), dense_shape=Tensor("PartitionedCall:5", shape=(3,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("PartitionedCall:1", shape=(3024,), dtype=int32), values=Tensor("Neg:0", shape=(3024, 3), dtype=float32), dense_shape=Tensor("PartitionedCall:2", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

      Step     Time          Energy         fmax
*Force-consistent energies used in optimization.
FIRE:    0 19:38:19     -279.345825*       8.6713   
C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/three_d_interaction_2/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/three_d_interaction_2/GatherV2_1_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/three_d_interaction_2/GatherV2_1_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_1_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_1_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.       

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_2/concat_atoms_2/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_2/concat_atoms_2/GatherV2_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_2/concat_atoms_2/GatherV2_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/three_d_interaction_1/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/three_d_interaction_1/GatherV2_1_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/three_d_interaction_1/GatherV2_1_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_1_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_1_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.       

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_1/concat_atoms_1/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_1/concat_atoms_1/GatherV2_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_1/concat_atoms_1/GatherV2_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/concat_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/concat:0", shape=(None,), dtype=float32), dense_shape=Tensor("gradients/m3g_net/three_d_interaction_2/GatherV2_2_grad/Cast:0", shape=(1,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/GatherV2_5_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/GatherV2_5_grad/Reshape:0", shape=(None,), dtype=float32), dense_shape=Tensor("gradients/m3g_net/GatherV2_5_grad/Cast:0", shape=(1,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/GatherV2_6_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/GatherV2_6_grad/Reshape:0", shape=(None,), dtype=float32), dense_shape=Tensor("gradients/m3g_net/GatherV2_6_grad/Cast:0", shape=(1,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/GatherV2_3_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/GatherV2_3_grad/Reshape:0", shape=(None, 3), dtype=float32), dense_shape=Tensor("gradients/m3g_net/GatherV2_3_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/GatherV2_4_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/GatherV2_4_grad/Reshape:0", shape=(None, 3), dtype=float32), dense_shape=Tensor("gradients/m3g_net/GatherV2_4_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("PartitionedCall:1", shape=(None,), dtype=int32), values=Tensor("Neg:0", shape=(None, 3), dtype=float32), dense_shape=Tensor("PartitionedCall:2", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

FIRE:    1 19:38:26     -274.974579*     150.3564
FIRE:    2 19:38:26     -269.860413*     163.3046
FIRE:    3 19:38:27     -278.717163*      61.7388
FIRE:    4 19:38:27     -279.168213*      36.7923
FIRE:    5 19:38:27     -279.419586*       4.7319
FIRE:    6 19:38:27     -279.420776*       4.4985
FIRE:    7 19:38:27     -279.423065*       4.0459
FIRE:    8 19:38:27     -279.426086*       3.4048
FIRE:    9 19:38:27     -279.429474*       2.6230
FIRE:   10 19:38:27     -279.432831*       2.2452
FIRE:   11 19:38:27     -279.435852*       1.8118
FIRE:   12 19:38:27     -279.438293*       1.3240
FIRE:   13 19:38:27     -279.440247*       1.2179
FIRE:   14 19:38:28     -279.441650*       1.7180
FIRE:   15 19:38:28     -279.442657*       1.9919
FIRE:   16 19:38:28     -279.442749*       1.9554
FIRE:   17 19:38:28     -279.442963*       1.8833
FIRE:   18 19:38:28     -279.443237*       1.7773
FIRE:   19 19:38:28     -279.443573*       1.6404
FIRE:   20 19:38:28     -279.443970*       1.4761
FIRE:   21 19:38:28     -279.444427*       1.2892
FIRE:   22 19:38:28     -279.444916*       1.0854
FIRE:   23 19:38:28     -279.445496*       0.8478
FIRE:   24 19:38:28     -279.446106*       0.5811
FIRE:   25 19:38:28     -279.446777*       0.3824
FIRE:   26 19:38:28     -279.447510*       0.3687
FIRE:   27 19:38:28     -279.448303*       0.3641
FIRE:   28 19:38:28     -279.449188*       0.5134
FIRE:   29 19:38:29     -279.450256*       0.6596
FIRE:   30 19:38:29     -279.451538*       0.7646
FIRE:   31 19:38:29     -279.453125*       0.7705
FIRE:   32 19:38:29     -279.455048*       0.6054
FIRE:   33 19:38:29     -279.457245*       0.3071
FIRE:   34 19:38:29     -279.459595*       0.3015
FIRE:   35 19:38:29     -279.462067*       0.5849
FIRE:   36 19:38:29     -279.464844*       0.6786
FIRE:   37 19:38:29     -279.467957*       0.4661
FIRE:   38 19:38:29     -279.471191*       0.2043
FIRE:   39 19:38:29     -279.474182*       0.4625
FIRE:   40 19:38:29     -279.477112*       0.4599
FIRE:   41 19:38:29     -279.479858*       0.2580
FIRE:   42 19:38:29     -279.482117*       0.5493
FIRE:   43 19:38:30     -279.484131*       0.2746
FIRE:   44 19:38:30     -279.485596*       0.6642
FIRE:   45 19:38:30     -279.486938*       0.5204
FIRE:   46 19:38:30     -279.487488*       1.7465
FIRE:   47 19:38:30     -279.488220*       0.3662
FIRE:   48 19:38:30     -279.488190*       1.2142
FIRE:   49 19:38:30     -279.488373*       0.9342
FIRE:   50 19:38:30     -279.488617*       0.4422
FIRE:   51 19:38:30     -279.488739*       0.1691
FIRE:   52 19:38:30     -279.488770*       0.5966
FIRE:   53 19:38:30     -279.488770*       0.5605
FIRE:   54 19:38:30     -279.488800*       0.4907
FIRE:   55 19:38:30     -279.488861*       0.3914
FIRE:   56 19:38:30     -279.488922*       0.2696
FIRE:   57 19:38:31     -279.488953*       0.1483
FIRE:   58 19:38:31     -279.489014*       0.1461
FIRE:   59 19:38:31     -279.489044*       0.1439
FIRE:   60 19:38:31     -279.489105*       0.2442
FIRE:   61 19:38:31     -279.489166*       0.3182
FIRE:   62 19:38:31     -279.489258*       0.3417
FIRE:   63 19:38:31     -279.489380*       0.3012
FIRE:   64 19:38:31     -279.489532*       0.1923
FIRE:   65 19:38:31     -279.489716*       0.1325
FIRE:   66 19:38:31     -279.489899*       0.1463
FIRE:   67 19:38:31     -279.490112*       0.2374
FIRE:   68 19:38:31     -279.490417*       0.1938
FIRE:   69 19:38:31     -279.490723*       0.1127
FIRE:   70 19:38:31     -279.491089*       0.1970
FIRE:   71 19:38:31     -279.491486*       0.2152
FIRE:   72 19:38:31     -279.491943*       0.0969
@shyuep
Copy link
Contributor

shyuep commented Jun 17, 2022

The tensorflow warnings can be captured with a catch warnings. But the FIRE: is output from ASE. I am trying to figure out how to control the verbosity, but I can't see an obvious way to do it (beyond supplying a log file, which I don't want to do).

@sgbaird
Copy link
Contributor Author

sgbaird commented Jun 17, 2022

For the FIRE:, if nothing on the ASE side, maybe filter out lines that begin with FIRE: and then reset the stdout after the function is called? Not the most elegant, but I verified it and it seems to works as expected:

import re


class Filter(object):
    """https://stackoverflow.com/a/63662744/13697228
    
    For reassigning default, https://stackoverflow.com/a/51340381/13697228"""

    def __init__(self, stream, re_pattern):
        self.stream = stream
        self.pattern = (
            re.compile(re_pattern) if isinstance(re_pattern, str) else re_pattern
        )
        self.triggered = False

    def __getattr__(self, attr_name):
        return getattr(self.stream, attr_name)

    def write(self, data):
        if data == "\n" and self.triggered:
            self.triggered = False
        else:
            if self.pattern.search(data) is None:
                self.stream.write(data)
                self.stream.flush()
            else:
                # caught bad pattern
                self.triggered = True

    def flush(self):
        self.stream.flush()
        if self.relax_on_decode:
            sys.stdout = Filter(sys.stdout, r"^FIRE:|     Step     Time          Energy         fmax|\*Force-consistent energies used in optimization.")  # type: ignore # noqa: E501
            relaxer = Relaxer()  # This loads the default pre-trained model

        # relaxation

        if self.relax_on_decode:
            # restore default https://stackoverflow.com/a/51340381/13697228
            sys.stdout = sys.__stdout__
2022-06-16 20:35:23.393379: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found  
2022-06-16 20:35:23.397911: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2022-06-16 20:35:23.405877: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("PartitionedCall:4", shape=(None,), dtype=int32), values=Tensor("PartitionedCall:3", shape=(None, 3, 3), dtype=float32), dense_shape=Tensor("PartitionedCall:5", shape=(3,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("PartitionedCall:1", shape=(3024,), dtype=int32), values=Tensor("Neg:0", shape=(3024, 3), dtype=float32), dense_shape=Tensor("PartitionedCall:2", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/three_d_interaction_2/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/three_d_interaction_2/GatherV2_1_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/three_d_interaction_2/GatherV2_1_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_1_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_2/gated_atom_update_2/GatherV2_1_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_2/concat_atoms_2/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_2/concat_atoms_2/GatherV2_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_2/concat_atoms_2/GatherV2_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/three_d_interaction_1/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/three_d_interaction_1/GatherV2_1_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/three_d_interaction_1/GatherV2_1_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_1_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_1/gated_atom_update_1/GatherV2_1_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer_1/concat_atoms_1/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer_1/concat_atoms_1/GatherV2_grad/Reshape:0", dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer_1/concat_atoms_1/GatherV2_grad/Cast:0", shape=(None,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/concat_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/concat:0", shape=(None,), dtype=float32), dense_shape=Tensor("gradients/m3g_net/three_d_interaction_2/GatherV2_2_grad/Cast:0", shape=(1,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/GatherV2_5_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/GatherV2_5_grad/Reshape:0", shape=(None,), dtype=float32), dense_shape=Tensor("gradients/m3g_net/GatherV2_5_grad/Cast:0", shape=(1,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.       

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/GatherV2_6_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/GatherV2_6_grad/Reshape:0", shape=(None,), dtype=float32), dense_shape=Tensor("gradients/m3g_net/GatherV2_6_grad/Cast:0", shape=(1,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.       

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/GatherV2_3_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/GatherV2_3_grad/Reshape:0", shape=(None, 3), dtype=float32), dense_shape=Tensor("gradients/m3g_net/GatherV2_3_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.     

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/GatherV2_4_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/GatherV2_4_grad/Reshape:0", shape=(None, 3), dtype=float32), dense_shape=Tensor("gradients/m3g_net/GatherV2_4_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.     

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("PartitionedCall:1", shape=(None,), dtype=int32), values=Tensor("Neg:0", shape=(None, 3), dtype=float32), dense_shape=Tensor("PartitionedCall:2", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/three_d_interaction/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/three_d_interaction/GatherV2_1_grad/Reshape:0", shape=(None, 9), dtype=float32), dense_shape=Tensor("gradients/m3g_net/three_d_interaction/GatherV2_1_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.   

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer/gated_atom_update/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer/gated_atom_update/GatherV2_grad/Reshape:0", shape=(None, 64), dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer/gated_atom_update/GatherV2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer/gated_atom_update/GatherV2_1_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer/gated_atom_update/GatherV2_1_grad/Reshape:0", shape=(None, 64), dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer/gated_atom_update/GatherV2_1_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.      

C:\Users\sterg\Miniconda3\envs\xtal2png-docs\lib\site-packages\tensorflow\python\framework\indexed_slices.py:444: UserWarning:

Converting sparse IndexedSlices(IndexedSlices(indices=Tensor("gradients/m3g_net/graph_network_layer/concat_atoms/GatherV2_grad/Reshape_1:0", shape=(None,), dtype=int32), values=Tensor("gradients/m3g_net/graph_network_layer/concat_atoms/GatherV2_grad/Reshape:0", shape=(None, 64), dtype=float32), dense_shape=Tensor("gradients/m3g_net/graph_network_layer/concat_atoms/GatherV2_grad/Cast:0", shape=(2,), dtype=int32))) to a dense Tensor of unknown shape. This may consume a large amount of memory.

@sgbaird
Copy link
Contributor Author

sgbaird commented Jun 17, 2022

For the tensorflow warnings, I went with:

import tensorflow as tf

...

if not self.verbose:
    tf.get_logger().setLevel(logging.ERROR)

which just leaves:

2022-06-16 20:48:47.442172: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'cudnn64_8.dll'; dlerror: cudnn64_8.dll not found
2022-06-16 20:48:47.446997: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1850] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2022-06-16 20:48:47.460638: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX AVX2To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.

I'm pretty sure this is just during the one call to Relax() (not the repeated calls to relaxer.relax()), and I'm fine with that.

@shyuep
Copy link
Contributor

shyuep commented Jun 17, 2022

I actually found a more elegant solution. Using contextlib.redirect_stdout....

@shyuep shyuep closed this as completed in a344a69 Jun 17, 2022
@shyuep
Copy link
Contributor

shyuep commented Jun 17, 2022

Yeah for the TF, I will leave them for now since it is relatively easy to silence by the end user if you wish. Tensorflow has a lot of deprecations and warnings and sometimes those are useful, especially for the developer end....

@sgbaird
Copy link
Contributor Author

sgbaird commented Jun 17, 2022

@shyuep great! That works for me. The contextlib solution is definitely a lot cleaner for the FIRE lines.

sgbaird added a commit to sparks-baird/xtal2png that referenced this issue Jun 17, 2022
sgbaird added a commit to sparks-baird/xtal2png that referenced this issue Jun 17, 2022
Can remove after next m3gnet release
materialsvirtuallab/m3gnet#11
@shyuep
Copy link
Contributor

shyuep commented Jun 17, 2022

Ok, I just released v0.06 that should have the verbosity control.

sgbaird added a commit to sparks-baird/xtal2png that referenced this issue Jun 17, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants