Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metrics chart wrongly connects points on X axis #45

Closed
zegerhoogeboom opened this issue Jun 14, 2018 · 3 comments · Fixed by #204
Closed

Metrics chart wrongly connects points on X axis #45

zegerhoogeboom opened this issue Jun 14, 2018 · 3 comments · Fixed by #204
Labels
bug Something isn't working

Comments

@zegerhoogeboom
Copy link

System information

  • Have I written custom code (as opposed to using a stock example script provided in MLflow): Yes
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
  • MLflow installed from (source or binary): source
  • MLflow version (run python -c "from mlflow import version; print(version.VERSION)"): 0.1.0
  • Python version: 3.6.5
  • npm version (if running the dev UI): 6.0.1
  • Exact command to reproduce: NA

Describe the problem

In the UI at the metrics graph view, it looks like this for me:
image

The data points are logged at the end of an epoch using this Keras callback:

class LossHistory(Callback):
    def on_epoch_end(self, epoch, logs=None):
        log_metric("loss", logs.get('loss'))
        # in reality more metrics here

logs.get('loss') just returns a number. The connections of the dots in the interface should be connecting X=0 to X=1 to X=2 and so on. Why they are currently connected in this strange way, I have no clue.

@mateiz
Copy link
Contributor

mateiz commented Jun 14, 2018

Is there any chance you could share your MLflow run folder that caused this (mlruns/0/<runID>) or the file for the metric inside there? Also, which browser are you using?

@mateiz mateiz assigned mateiz and andrewmchen and unassigned mateiz Jun 18, 2018
@zegerhoogeboom
Copy link
Author

zegerhoogeboom commented Jun 19, 2018

Terribly sorry for the late reply. cat mlruns/0/2924aa6b449944bc849d4381e0fa3570/metrics/loss:

0 4.54306590325
0 3.50579357207
0 3.63842143944
0 3.63074730358
0 3.62895112526
0 3.60019535707
0 3.34938469377
0 3.69437173072
0 3.4442458695
0 5.05011896209

cat mlruns/0/2924aa6b449944bc849d4381e0fa3570/meta.yml:

artifact_uri: /app/mlruns/0/2924aa6b449944bc849d4381e0fa3570/artifacts
end_time: null
entry_point_name: !!python/unicode ''
experiment_id: !!python/long '0'
name: Run 5
run_uuid: 2924aa6b449944bc849d4381e0fa3570
source_name: !!python/unicode '/home/{my source code location}.py'
source_type: 4
source_version: !!python/unicode '94de7c2a7562525db2ead17fe8d020a7ca1cf32a'
start_time: !!python/long '1529310274917'
status: 1
tags: []
user_id: !!python/unicode '{my user name}'

It's being run from a Docker container using continuumio/miniconda, the host is running CentOS 7. I'm using Chrome (version 67) as my browser.

Note that the experiment in my screenshot and this one are not the same but the same happens for any experiment.

@mateiz
Copy link
Contributor

mateiz commented Jun 21, 2018

Interesting, I don't see this on Chrome 67 with the same file. Do you have any extensions that might be blocking JavaScript? It's also a bit weird that your metrics are being logged with 0 as the timestamp (it should be the time since the epoch), but I tried it with either 0 or a recent timestamp and it seems to work OK.

@mateiz mateiz added the bug Something isn't working label Jun 28, 2018
@mateiz mateiz closed this as completed Jun 28, 2018
@mateiz mateiz reopened this Jun 28, 2018
mateiz added a commit that referenced this issue Jul 30, 2018
This fixes #45, which happens because the JavaScript sorting code we had for combining metrics would sometimes reorder them based on the order of fields in some objects (especially when metrics had the same timestamp).
aarondav pushed a commit to aarondav/mlflow that referenced this issue Jul 31, 2018
This fixes mlflow#45, which happens because the JavaScript sorting code we had for combining metrics would sometimes reorder them based on the order of fields in some objects (especially when metrics had the same timestamp).
jdlesage pushed a commit to jdlesage/mlflow that referenced this issue Dec 23, 2019
* added flask endpoints serving private vcs info to UI, displaying private vcs links in mflow runs table
sunishsheth2009 pushed a commit that referenced this issue Jun 16, 2022
Make pipelines test path configurable
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants