Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a tutorial to load metrics from prometheus server #774

Merged
merged 13 commits into from Feb 14, 2020

Conversation

@yongtang
Copy link
Member

yongtang commented Feb 7, 2020

This PR adds tutorial to load metrics from prometheus server
into tf.data.Dataset, so that it could be used for tf.keras
and model training.

Signed-off-by: Yong Tang yong.tang.github@outlook.com

@review-notebook-app

This comment has been minimized.

Copy link

review-notebook-app bot commented Feb 7, 2020

Check out this pull request on  ReviewNB

You'll be able to see Jupyter notebook diff and discuss changes. Powered by ReviewNB.

This PR adds tutorial to load metrics from prometheus server
into tf.data.Dataset, so that it could be used for tf.keras
and model training.

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
@yongtang yongtang force-pushed the yongtang:prometheus branch from 6ff78eb to 63d495c Feb 13, 2020
@yongtang yongtang marked this pull request as ready for review Feb 13, 2020
@yongtang

This comment has been minimized.

Copy link
Member Author

yongtang commented Feb 13, 2020

This is another tutorial to load metrics data from prometheus server. prometheus is almost a common setup in kubernetes, and is widely seem in cloud-native devops/infrastructure. this tutorial could help devops/sre to use tensorflow/tensorflow-io to automate their operations.

@yongtang yongtang requested review from MarkDaoust and lamberta Feb 13, 2020
@yongtang

This comment has been minimized.

Copy link
Member Author

yongtang commented Feb 13, 2020

/cc @MarkDaoust @lamberta to take a look.

docs/tutorials/prometheus.ipynb Outdated Show resolved Hide resolved
"metadata": {
"id": "yZmI7l_GykcW",
"colab_type": "text"
},

This comment has been minimized.

Copy link
@lamberta

lamberta Feb 13, 2020

Member

I'd create another code cell here that imports all the modules below. This way they're all in one place in the setup section.

docs/tutorials/prometheus.ipynb Outdated Show resolved Hide resolved
"\n",
"# Run `./coredns` as a background process.\n",
"# IPython doesn't recognize `&` in inline bash cells.\n",
"get_ipython().system_raw('./coredns &')"

This comment has been minimized.

Copy link
@lamberta

lamberta Feb 13, 2020

Member

Use separate code cells for shell commands and python. We have styles that make this clearer when imported into the website.

docs/tutorials/prometheus.ipynb Outdated Show resolved Hide resolved
docs/tutorials/prometheus.ipynb Outdated Show resolved Hide resolved
docs/tutorials/prometheus.ipynb Outdated Show resolved Hide resolved
"from datetime import datetime\n",
"import tensorflow_io as tfio\n",
"\n",
"dataset = tfio.experimental.IODataset.from_prometheus(\n",

This comment has been minimized.

Copy link
@lamberta

lamberta Feb 13, 2020

Member

Getting an error here when run on Colab. Is this expected?

@MarkDaoust Since this notebook is spinning up servers and such, will this work when imported into tensorflow.org? Do we need to save the outputs?

Trace:

WARNING:tensorflow:AutoGraph could not transform <function io_prometheus_readable_read at 0x7f5de5f60e18> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Unreachable
WARNING: AutoGraph could not transform <function io_prometheus_readable_read at 0x7f5de5f60e18> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Unreachable
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-6-5ea8e8be172d> in <module>()
      3 
      4 dataset = tfio.experimental.IODataset.from_prometheus(
----> 5     "coredns_dns_request_count_total", 5)
      6 
      7 print("Dataset Spec:\n{}\n".format(dataset.element_spec))

12 frames
/tensorflow-2.1.0/python3.6/tensorflow_core/python/autograph/impl/api.py in wrapper(*args, **kwargs)
    235       except Exception as e:  # pylint:disable=broad-except
    236         if hasattr(e, 'ag_error_metadata'):
--> 237           raise e.ag_error_metadata.to_exception(e)
    238         else:
    239           raise

AssertionError: in converted code:

    /usr/local/lib/python3.6/dist-packages/tensorflow_io/core/python/ops/prometheus_dataset_ops.py:82 f  *
        timestamp, value = golang_ops.io_prometheus_readable_read(
    <string>:254 io_prometheus_readable_read
        
    /tensorflow-2.1.0/python3.6/tensorflow_core/python/framework/op_def_library.py:602 _apply_op_helper
        assert False, "Unreachable"

    AssertionError: Unreachable

This comment has been minimized.

Copy link
@MarkDaoust

MarkDaoust Feb 13, 2020

Contributor

Since this notebook is spinning up servers and such, will this work when imported into tensorflow.org?

I think this part shouldn't be a problem.

MarkDaoust and others added 7 commits Feb 13, 2020
+ sudo apt
+ split dig commands.
Co-Authored-By: Billy Lamberta <blamb@google.com>
Co-Authored-By: Billy Lamberta <blamb@google.com>
Co-Authored-By: Billy Lamberta <blamb@google.com>
Co-Authored-By: Billy Lamberta <blamb@google.com>
Co-Authored-By: Billy Lamberta <blamb@google.com>
"from datetime import datetime\n",
"import tensorflow_io as tfio\n",
"\n",
"dataset = tfio.experimental.IODataset.from_prometheus(\n",

This comment has been minimized.

Copy link
@MarkDaoust

MarkDaoust Feb 13, 2020

Contributor

Since this notebook is spinning up servers and such, will this work when imported into tensorflow.org?

I think this part shouldn't be a problem.

MarkDaoust and others added 5 commits Feb 13, 2020
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Fix typo
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
…also split shell and python block

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
… (server may take a second or two to start up)

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
@yongtang

This comment has been minimized.

Copy link
Member Author

yongtang commented Feb 14, 2020

Thanks @lamberta @MarkDaoust for the review. I have updated the PR to place import modules into one section, and split the python and shell code, as was suggested in the review.

I also move the server setup into an early section, that should solve the potential AssertionError: Unreachable issue. The issue is likely caused by server not ready yet (may take a second or two to setup.)

Please take a look and let me know if there are any other issues.

Copy link
Member

lamberta left a comment

Thanks. Real test will be when we import (and execute) the notebooks to generate the webpage 😉

@yongtang yongtang merged commit 37ea2d6 into tensorflow:master Feb 14, 2020
19 checks passed
19 checks passed
Lint
Details
Bazel macOS
Details
Bazel Linux
Details
Bazel Windows
Details
Wheel 3.5 macOS
Details
Wheel 3.6 macOS
Details
Wheel 3.7 macOS
Details
Wheel 3.5 Linux
Details
Wheel 3.6 Linux
Details
Wheel 3.7 Linux
Details
Wheel 3.5 Windows
Details
Wheel 3.6 Windows
Details
Wheel 3.7 Windows
Details
Ubuntu GPU Python2 Internal CI build successful
Details
Ubuntu GPU Python3 Internal CI build successful
Details
Ubuntu Python2 Internal CI build successful
Details
Ubuntu Python3 Internal CI build successful
Details
Ubuntu Sanity Check Internal CI build successful
Details
cla/google All necessary CLAs are signed
@yongtang yongtang deleted the yongtang:prometheus branch Feb 14, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.