Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Running make on linux complains about pip #14

Closed
slochower opened this issue Sep 11, 2018 · 30 comments
Closed

Running make on linux complains about pip #14

slochower opened this issue Sep 11, 2018 · 30 comments

Comments

@slochower
Copy link

I'm trying to install and run the package on linux in a Python 2.7 conda environment, and I'm getting stuck at make setup. I see an error for pip (copied below), but I'm not sure what's going on because pip is definitely installed in this environment. I also manually installed sqlalchemy with conda to see if this would skip that part of make (it doesn't). Can you suggest any debugging tips?

...
Installed /usr/lib/python2.7/site-packages/python_tcxparser-0.7.2-py2.7.egg
Processing dependencies for python-tcxparser==0.7.2
Searching for lxml==3.2.1
Best match: lxml 3.2.1
Adding lxml 3.2.1 to easy-install.pth file

Using /usr/lib64/python2.7/site-packages
Finished processing dependencies for python-tcxparser==0.7.2
writing list of installed files to 'files.txt'
sudo pip install --upgrade sqlalchemy
sudo: pip: command not found
make: *** [deps] Error 1

...
(py27) Tue Sep 11 07:52:31 warthog:~/data/explorations/GarminDB
$ pip --version
pip 18.0 from /home/XXX/data/applications/anaconda3/envs/py27/lib/python2.7/site-packages/pip (python 2.7)
@tcgoetz
Copy link
Owner

tcgoetz commented Sep 11, 2018

pip is installed for your user. sudo is using user 'root''s environment and I bet pip is not in root's path. I assume sudo pip --version also fails.

Sicne you have python installed for your user only, you don't want sudo pip, just pip. I can look later at making it configurable to install python packages for all users for the user. In the meantime you can remove 'sudo' form in front of pip.

@slochower
Copy link
Author

Oh, good catch. Yes, that's correct. I'm running inside a conda environment because my system Python is 3.6. I removed the sudo lines in the Makefile, and then had to do a git add . and git commit to get the subsequent git pull for a submodule to work.

@slochower
Copy link
Author

It appears that the HealthData/Sleep directory isn't created. After I successfully get through make setup, when I try make GC_DATE=<date to start downloading data from> GC_DAYS={number of days of data to download} GC_USER={username} GC_PASSWORD={password} build_dbs, I get:

python analyze_garmin.py -S22:00,06:00  --sqlite /Users/tgoetz/HealthData/DBs
analyze_garmin.py -s <sqlite db path> -m ...
python import_garmin.py -e --sleep_input_dir "/home/XXX/HealthData/Sleep" --sqlite /home/XXX/HealthData/DBs
INFO:import_garmin.py:Debug: False
INFO:/home/XXX/hgst-3tb-data/explorations/GarminDB/FileProcessor.py:Reading directory: /home/XXX/HealthData/Sleep
Traceback (most recent call last):
  File "import_garmin.py", line 307, in <module>
    main(sys.argv[1:])
  File "import_garmin.py", line 296, in main
    gsd = GarminSleepData(sleep_input_file, sleep_input_dir, latest, debug)
  File "import_garmin.py", line 98, in __init__
    self.file_names = FileProcessor.FileProcessor.dir_to_files(input_dir, 'sleep_.*\.json', latest)
  File "/home/XXX/hgst-3tb-data/explorations/GarminDB/FileProcessor.py", line 28, in dir_to_files
    for file in os.listdir(input_dir):
OSError: [Errno 2] No such file or directory: '/home/XXX/HealthData/Sleep'
make: *** [import_sleep] Error 1

(py27) Tue Sep 11 08:12:47 warthog:~/data/explorations/GarminDB
$ ls ~/HealthData/
DBs  FitFiles

@tcgoetz
Copy link
Owner

tcgoetz commented Sep 13, 2018 •
edited
Loading

commit 95a1857 (HEAD -> master, origin/master, origin/HEAD)
Author: Tom Goetz <******>
Date: Thu Sep 13 07:38:55 2018 -0400

Issue #11: make installing python deps as root optional; fix make dep for creating sleep dir

@slochower
Copy link
Author

Thanks for the update! This gets me past the initial hurdle and it started to download a bunch of sleep data. However, when trying to import it (I think), I think epoc_ms is set to None.'

INFO:import_garmin.py:Importing /home/XXX/HealthData/Sleep/sleep_2018-07-10.json (2018-07-10) without REM data
INFO:import_garmin.py:DB updated 2018-07-10 with 0 sleep level entries
Traceback (most recent call last):
  File "import_garmin.py", line 307, in <module>
    main(sys.argv[1:])
  File "import_garmin.py", line 298, in main
    gsd.process_files(db_params_dict)
  File "import_garmin.py", line 126, in process_files
    json_data = json.load(open(file_name), object_hook=json_parser)
  File "/home/XXX/data/applications/anaconda3/envs/py27/lib/python2.7/json/__init__.py", line 291, in load
    **kw)
  File "/home/XXX/data/applications/anaconda3/envs/py27/lib/python2.7/json/__init__.py", line 352, in loads
    return cls(encoding=encoding, **kw).decode(s)
  File "/home/XXX/data/applications/anaconda3/envs/py27/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/XXX/data/applications/anaconda3/envs/py27/lib/python2.7/json/decoder.py", line 380, in raw_decode
    obj, end = self.scan_once(s, idx)
  File "import_garmin.py", line 111, in json_parser
    entry['sleepStartTimestampGMT'] = Fit.Conversions.epoch_ms_to_dt(entry['sleepStartTimestampGMT'])
  File "/home/XXX/hgst-3tb-data/explorations/GarminDB/Fit/Conversions.py", line 62, in epoch_ms_to_dt
    return datetime.datetime.fromtimestamp(epoch_ms / 1000.0)
TypeError: unsupported operand type(s) for /: 'NoneType' and 'float'
make: *** [import_sleep] Error 1

After doing a git pull and stashing my local changes, I just tried running make GC_USER={} GC_PASSWORD={} build_dbs. Is that correct?

tcgoetz added a commit that referenced this issue Sep 14, 2018
@tcgoetz
Issue #14: check for None when converting JSON sleep values
98be89c
@tcgoetz
Copy link
Owner

tcgoetz commented Sep 14, 2018

If you look at 2018-07-10 on the Garmin site, do you have sleep data? Should there be 0 entries for that date?

INFO:import_garmin.py:DB updated 2018-07-10 with 0 sleep level entries

@slochower
Copy link
Author

Nope -- no sleep data for that day (which is correct).

@tcgoetz
Copy link
Owner

tcgoetz commented Sep 16, 2018

Is it working without issues now?

@slochower
Copy link
Author

Getting a little farther. I just did a pull and re-ran make. I think now it didn't make the directory MSHealth.

INFO:analyze_garmin.py:Monitoring Years (0): []
python import_mshealth_csv.py -e --input_dir "/home/XXX/HealthData/MSHealth" --sqlite /home/XXX/HealthData/DBs
Traceback (most recent call last):
  File "import_mshealth_csv.py", line 180, in <module>
    main(sys.argv[1:])
  File "import_mshealth_csv.py", line 170, in main
    msd = MSHealthData(input_file, input_dir, db_params_dict, english_units, debug)
  File "import_mshealth_csv.py", line 65, in __init__
    self.file_names = FileProcessor.FileProcessor.dir_to_files(input_dir, 'Daily_Summary_.*.csv')
  File "/home/XXX/hgst-3tb-data/explorations/GarminDB/FileProcessor.py", line 28, in dir_to_files
    for file in os.listdir(input_dir):
OSError: [Errno 2] No such file or directory: '/home/XXX/HealthData/MSHealth'
make: *** [import_mshealth] Error 1

I certainly appreciate the support, but don't feel any time pressure to work on this for me. I'm just interested in playing with the data when I have it.

@tcgoetz
Copy link
Owner

tcgoetz commented Sep 16, 2018

commit e0f7314 (HEAD -> master, origin/master, origin/HEAD)

Author: Tom Goetz xxxxxxxx@gmail.com
Date: Sun Sep 16 13:54:24 2018 -0400

Issues: 14: create mshealth and fitbit dirs if needed

NP. Your debugging it for me on a platform I'm not using, so your helping me.

@slochower
Copy link
Author

It appears to finish make!

Although I'm not sure what things are in which database. If I open garmin.db in SQLiteStudio I can see "resting_hr" under Tables and the Structure looks good (index 1 is "day" and index 2 is "resting_heart_rate") but there is nothing under the Data tab. garmin.db is 164 kb; I'm not sure if that is too small.

@tcgoetz
Copy link
Owner

tcgoetz commented Sep 16, 2018

Is there data in the JSON files in HealthData/RHR/ ?

@slochower
Copy link
Author

slochower commented Sep 16, 2018 via email

@tcgoetz
Copy link
Owner

tcgoetz commented Sep 16, 2018

What types of data do you have on Connect? Do the dates you have data for fall in the values you used for GC_DATE and GC_DAYS?

You can try just downloading particular types of data to see if you get what you expect:

make GC_USER={username} GC_PASSWORD={password} download_monitoring
make GC_USER={username} GC_PASSWORD={password} download_all_activities
make GC_USER={username} GC_PASSWORD={password} download_sleep
make GC_USER={username} GC_PASSWORD={password} download_rhr

You can also try running

make GC_USER={username} GC_PASSWORD={password}

and it should download data for the previous year.

@slochower
Copy link
Author

I tried your last command to get all the data for the previous year. STill gives None for some things that have recorded in Connect. Ohhhhhh. I tried to login to the website -- "Your account has been locked. Please reset your password." I did reset the password now, but I'm wary of running make, in case it gets locked again. Any suggestions?

@tcgoetz
Copy link
Owner

tcgoetz commented Sep 18, 2018

Actually, no, I've never had my account get locked, even in the early development when I was working out the login.

tcgoetz added a commit that referenced this issue Sep 18, 2018
@tcgoetz
Issue #14: reduce default period for getting data from one year to on…
8cce566 
…e month to avoid flooding
@tcgoetz
Copy link
Owner

tcgoetz commented Sep 18, 2018

Thinking about this more, I have a guess. The sleep, RHR, and a few other types of data can get a whole year+ of data from one graph. That's one page access. Monitoring data is retrieved one page access per day. Accessing 365 monitoring pages in a row probably set off a threshold alarm. I've chnaged the default to get one month of data. Hopefully that's small enough. To get your data set you can get it one month at a time with sequential fetches spaced out over time using GC_DATE and GC_DAYS to cover the period you need. I'll look at adding some pauses into the web browsing.

tcgoetz added a commit that referenced this issue Sep 18, 2018
@tcgoetz
Issue #14: pause for one second between fetching of monitoring days t…
6e7b5a6 
…o avoid flooding
tcgoetz added a commit that referenced this issue Sep 18, 2018
@tcgoetz
Issue #14: pause for one second between fetching of sleep chunks, rhr…
4d7f705 
… chunks, and activity files to avoid flooding
@tcgoetz
Copy link
Owner

tcgoetz commented Sep 25, 2018

Have you tried this again? Does this work for you?

@slochower
Copy link
Author

slochower commented Sep 25, 2018 via email

@slochower
Copy link
Author

slochower commented Oct 1, 2018 •
edited
Loading

Hm, just did a make pull and then make GC_USER=xxxxxxxxx GC_password=xxxxxx build_dbs and I see

python analyze_garmin.py -S22:00,06:00 --sqlite /Users/tgoetz/HealthData/DBs
Traceback (most recent call last):
  File "analyze_garmin.py", line 10, in <module>
    import GarminDB
  File "/home/dslochower/hgst-3tb-data/explorations/GarminDB/GarminDB/__init__.py", line 1, in <module>
    from GarminDB import *
  File "/home/dslochower/hgst-3tb-data/explorations/GarminDB/GarminDB/GarminDB.py", line 8, in <module>
    from Fit import FieldEnums
ImportError: cannot import name FieldEnums
make: *** [garmin_config] Error 1

@tcgoetz
Copy link
Owner

tcgoetz commented Oct 1, 2018

There is no "make pull". Did you mean "git pull" or "make update"? Either way it looks like your Fit submodule didn't get updated.

@slochower
Copy link
Author

Oops, git pull. But you're right -- doing git submodule update pulls in a new Fit submodule.

Now seeing:

Traceback (most recent call last):
  File "import_garmin.py", line 301, in <module>
    main(sys.argv[1:])
  File "import_garmin.py", line 292, in main
    gsd.process_files(db_params_dict)
  File "import_garmin.py", line 110, in process_files
    garmindb = GarminDB.GarminDB(db_params_dict)
  File "/home/dslochower/hgst-3tb-data/explorations/GarminDB/GarminDB/GarminDB.py", line 27, in __init__
    self.version.version_check(self, self.db_version)
  File "/home/dslochower/hgst-3tb-data/explorations/GarminDB/HealthDB/DB.py", line 533, in version_check
    raise RuntimeError("DB %s version mismatch. Please rebuild the DB. (%s vs %s)" % (db.db_name, self.version, version_number))
RuntimeError: DB garmin version mismatch. Please rebuild the DB. (1 vs 3)
make: *** [import_sleep] Error 1

@tcgoetz
Copy link
Owner

tcgoetz commented Oct 1, 2018

Yep. See the first bullet in the "Notes" section of the front page.

@slochower
Copy link
Author

Awesome! I opened garmin.db and I see data in sleep and sleep_events but not the other categories.

@tcgoetz
Copy link
Owner

tcgoetz commented Oct 2, 2018 •
edited
Loading

Sleep data comes from json files downloaded into ~/HealthData/Sleep/. The other columns come from daily monitoring fit files downloaded into ~/HealthData/FitFiles/_Monitoring. Do you have that directory and are there files in it? On the Garmin Connect website what date ranges do you have "Daily Summary" pages for? Are those same date ranges represented in ~/HealthData/FitFiles/_Monitoring? If not then you need to download them via "make GC_DATE=[date to start downloading] GC_DAYS=[number of days to download] download_monitoring"

@slochower
Copy link
Author

Ah. I do have FitFiles and 2018_Monitoring but nothing inside of them. On Garmin connect, under "Daily Summary" I don't have anything for today, but I do have yesterday, and the day before that, etc. Not sure how to find the overall date range.

@tcgoetz
Copy link
Owner

tcgoetz commented Oct 2, 2018

Actually, there is an easier way to tell how much and what kind of data you have. When you run "make" and you DB is updated, the last thing that does it to run "make garmin_summary" which displays stats about whats in the DB.

Here's the summary Re monitoring data I get:

INFO:analyze_garmin.py:Monitoring records: 224701
INFO:analyze_garmin.py:Monitoring Years (2): [2017, 2018]
INFO:analyze_garmin.py:2017 Months (11): ['Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep', 'Oct', 'Nov', 'Dec']
INFO:analyze_garmin.py:2017 Days (309 count vs 310 span): [56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299, 300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313, 314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341, 342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355, 356, 357, 358, 359, 360, 361, 362, 363, 364, 365]
INFO:analyze_garmin.py:Days gap between 78 (2017-03-19 00:00:00) and 80 (2017-03-21 00:00:00)
INFO:analyze_garmin.py:2018 Months (9): ['Jan', 'Feb', 'Mar', 'Apr', 'May', 'Jun', 'Jul', 'Aug', 'Sep']
INFO:analyze_garmin.py:2018 Days (273 count vs 273 span): [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273]

@tcgoetz
Copy link
Owner

tcgoetz commented Oct 2, 2018

Why don't you start with downloading the last months data? What happens when you run:
make download_monitoring

@slochower
Copy link
Author

Weird. Getting a login error even though I just confirmed via connect.garmin.com it works. I'll look into it further tonight or tomorrow.

One thought: I don't see anything after -p here. Is it possible that it is not reading GC_password environmental variable?

python download_garmin.py -d 09/01/2018 -n 31 -u slochower@gmail.com -p  -m "/home/dslochower/HealthData/FitFiles/2018_Monitoring"
ERROR:root:Login failed: <!DOCTYPE html>

@tcgoetz
Copy link
Owner

tcgoetz commented Oct 2, 2018

It's GC_PASSWORD. The case matters.

@tcgoetz tcgoetz closed this as completed Jan 30, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@slochower @tcgoetz and others