Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stats.log file is empty, yet weighs in at 500+ meg #62

Open
OniDaito opened this issue May 1, 2020 · 0 comments
Open

stats.log file is empty, yet weighs in at 500+ meg #62

OniDaito opened this issue May 1, 2020 · 0 comments

Comments

@OniDaito
Copy link

OniDaito commented May 1, 2020

Hello there. I'm trying to load a log file, saved with tensorwatch.
I have a machine at the end of the internet, logging to a file. When a run is completed I upload the log file to a server, where a jupyter notebook is running. My hope is to open that log file and do all my work there where it can be shared.
Eventually, I'd like to stream my results but that requires a lot more work with tunnelling and security. For now, this method would be fine.

On the AI box:


class Stats(object):
    """ We used a stats object as we have streams and all that sort of
     thing."""

    def __init__(self):
        # create a stream for logging
        self.watching = {}
        self.observing = {}

    def on(self, savedir, port):
        self.watcher = tw.Watcher(filename=savedir + "/stats.log", port=port)
        # self.watcher.make_notebook(savedir + "/stats.ipynb")

    def watch(self, obj, name):
        """ Add something to be watched via tensorwatch. """
        if name not in self.watching.keys():
            self.watching[name] = {
                "object": obj,
                "stream": self.watcher.create_stream(name=name)}

    def observe(self, obj, name):
        """ Add something to be observed with tensorwatch. This is apparently
        low cost and lazy and I suspect only happens when we actually look
        at things live. Could be very handy though. """
        if name not in self.observing.keys():
            kw = {name: obj}
            self.observing[name] = obj
            self.watcher.observe(**kw)

    def update(self, idx):
        """ Update all our streams with the current idx value. """
        for name in self.watching.keys():
            dd = self.watching[name]
            dd['stream'].write((idx, dd['object']))

I use watch to add objects such as tensors with a label. Update is called when I start writing to the stream

On the jupyter notebook side I have this:

import tensorwatch as tw
import pickle
import sys
import torch

client = tw.WatcherClient(port=None, filename="stats.log")
in_e = client.open_stream("in_e")
out_e = client.open_stream("out_e")

text_vis = tw.Visualizer(out_e, vis_type='text')

The text_vis loads (with a bit of extra coddling to convert from GPU to CPU tensors) but it's empty! 0 entries.

Is there something I'm doing wrong here? Ideally, I'd like to see a tutorial on loading from a file or interrogating these files so I can manipulate the data stored in there.

I'm using Python 3.7 with pytorch 1.5.0

Cheers
Ben

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant