Possible date conversion error? #1
Comments
|
Thanks for taking the time to open an issue! Can you give me more information about your environment? That will help me diagnose the problem. Specifically: the versions of your operating system, Python, pyculiarity, its dependencies (SciPy, NumPy, R, nose, pip), etc. Also, how did you install the package?The more detail the better. |
|
Sure thing, would love to help on this. I started the install by downloading the .zip file (right side of https://github.com/nicolasmiller/pyculiarity), and unzipped. To answer your questions: Please let me know if you need more info. I have a log.txt that shows all my steps, though do not know how to attach to this comment; If you would like to see, please let me know how to send to you/post here. I am going to play around with etsy/skyline for my current project, but would love to come back to pyculiarity to supplement. Thanks! |
|
Ah, I should have noticed Win32 in the stacktrace. The package has been tested exclusively in *nix environments, both OS X and Linux (this should be more explicit in the documentation). I'll need to scrounge up a Windows box to test on. I should be able to take a look later this evening. |
|
Thanks. As a theoretical question, why would the OS environment matter for TypeErrors? To me, this sounds like an environment-independent issue dealing with a math operation on strings, rather than ints/floats, no? |
|
I would tend to agree with you just looking at the stacktrace. But I know the example works just fine in other environments. There might be some slightly different behavior happening in your versions of the dependencies for example. I want to see if I can reproduce your exact problem and then work backwards from there. Oh, can you give me your versions of pandas, pytz, statsmodels and rpy2 as well? |
|
But, yes, it looks like we do end up with string dates where the code is expecting actual date objects that can be subtracted. So for some reason the normal conversion that should happen before we reach that point is failing. I'll keep you posted with what I find once I have a machine to debug with. In the meantime, it might be worth tracing through what's happening in your environment around the format_timestamp function to understand why exactly the conversion from a date string isn't happening. |
|
pandas: 0.13.1 |
|
perhaps this is overkill, but here is the log. Feel free to delete this comment if excessive :) Microsoft Windows [Version 6.1.7601] C:\Python27\Lib\site-packages\pyculiarity-master> C:\Python27\Lib\site-packages\pyculiarity-master> Installed c:\python27\lib\site-packages\pyculiarity-0.0.1a3-py2.7.egg Using c:\python27\lib\site-packages Using c:\python27\lib\site-packages Using c:\python27\lib\site-packages Using c:\python27\lib\site-packages Using c:\python27\lib\site-packages Using c:\python27\lib\site-packages Using c:\python27\lib\site-packages Using c:\python27\lib\site-packages Using c:\python27\lib\site-packages C:\Python27\Lib\site-packages\pyculiarity-master> EEEEEEEEEEEEE ERROR: test_check_constant_series (test_edge.TestEdge) Traceback (most recent call last): ERROR: test_check_midnight_date_format (test_edge.TestEdge) Traceback (most recent call last): ERROR: test_handling_of_leading_trailing_nas (test_na.TestNAs) Traceback (most recent call last): ERROR: test_handling_of_middle_nas (test_na.TestNAs) Traceback (most recent call last): ERROR: test_both_directions_e_value_longterm (test_ts.TestTS) Traceback (most recent call last): ERROR: test_both_directions_e_value_threshold_med_max (test_ts.TestTS) Traceback (most recent call last): ERROR: test_both_directions_with_plot (test_ts.TestTS) Traceback (most recent call last): ERROR: test_both_directions_e_value_longterm (test_ts_AR.TestTS) Traceback (most recent call last): ERROR: test_both_directions_e_value_threshold_med_max (test_ts_AR.TestTS) Traceback (most recent call last): ERROR: test_both_directions_with_plot (test_ts_AR.TestTS) Traceback (most recent call last): ERROR: test_both_directions_e_value_longterm (test_vec.TestVec) Traceback (most recent call last): ERROR: test_both_directions_e_value_threshold_med_max (test_vec.TestVec) Traceback (most recent call last): ERROR: test_both_directions_with_plot (test_vec.TestVec) Traceback (most recent call last): Ran 13 tests in 6.751s FAILED (errors=13) C:\Python27\Lib\site-packages\pyculiarity-master> |
|
Excellent! This really helps. |
|
I managed to get a Windows environment up on my Mac under Parallels and have reproduced the error. Working on a fix now. Another thing occurred to me: are you on 32-bit or 64-bit Windows? And are you using 32 or 64 bit versions of the python dependencies? |
|
Glad you could reproduce and that it wasn't some random issue with my configuration. Progress! i am on 64-bit Windows. I am not sure about 32 vs 64 bit Python dependencies; how can I check? I am running my scripts out of Python(x.y), FYI. |
|
I will also dig into the issue today on my end, and see what i can find. Please let me know how i can assist. Looking forward to getting this up and running. |
|
I managed to get it working on Windows with all of your versions of the dependencies save one: statsmodels. I needed to upgrade to 0.6.1 from 0.5.0. Between those versions, there were changes made to statsmodels.robust.scale.mad (median absolute deviation) which we use in detect_anoms. This was throwing results off slightly. I had to make a couple of other minor changes to accommodate for some other differences with some of the numpy and pandas types. Take a look at the commit history if you're interested. Long story short: grab the latest code from master and upgrade to at least statsmodels 0.6.1 and give it a shot. If you're stuck with 0.5.0 for some reason, you'll need to replace the mad function in detect_anoms with something that is consistent with what 0.6.1's mad returns. |
|
Excellent!! Nice work! I will give this a shot today, and reply with my results. |
|
@nicolasmiller, I am still having issues :(. As instructed, I grabbed the latest statsmodels and pyculiarity codes, installed, getting similar (maybe same) error as before. When I check versions, I get:
i have tried a simple example, but it gives me an error: print "(largest - second_largest): ", (np.timedelta64(largest) - np.timedelta64(second_largest)) ValueError: Could not convert object to NumPy timedelta however,
I am not sure of what this indicates. I think that it has something to do with the string including the days and hour/min/sec all in one string, which perhaps numpy doesn't like. I will think more tomorrow, but just wanted to give you a heads up. Any suggestions what I am doing wrong is appreciated. |
|
Sorry to hear that. I'll take another look. There must still be something different about our environments and dependencies, etc. Can you double check all of the version numbers and confirm that they're still the same as what you listed before? |
|
oh no, sorry about the late reply - I am just seeing this now. I should have some time tomorrow to send you the info that you requested above, as well as any diagnostics that I can come up with. Additionally, I will test on my home Windows machine (Windows 10) to see if this is just some strange occurrence on my production machine. Thanks for your efforts! |
Hello,
I am trying to run the example that you have posted on the main page, ie:
from pyculiarity import detect_ts
import pandas as pd
twitter_example_data = pd.read_csv('raw_data.csv',
usecols=['timestamp', 'count'])
results = detect_ts(twitter_example_data,
max_anoms=0.02,
direction='both', only_last='day')
but unfortunately, i get an error (please see below):
Traceback (most recent call last):
File "", line 1, in
File "C:\Python27\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 540, in runfile
execfile(filename, namespace)
File "C:/Users/user1/Current projects//test/test_pyculiarity_0.py", line 7, in
direction='both', only_last='day')
File "build\bdist.win32\egg\pyculiarity\detect_ts.py", line 142, in detect_ts
gran = get_gran(df)
File "build\bdist.win32\egg\pyculiarity\date_utils.py", line 46, in get_gran
gran = int(round((largest - second_largest) / np.timedelta64(1, 's')))
TypeError: ufunc divide cannot use operands with types dtype('O') and dtype('<m8[s]')
i have traced the issue best that i could, and it appears that there is an error with the "largest - second_largest" step:
14395 1980-10-05 13:56:00
14396 1980-10-05 13:57:00
14397 1980-10-05 13:58:00
Name: timestamp, Length: 14398, dtype: object
nlargest(2, col): ['1980-10-05 13:58:00', '1980-10-05 13:57:00']
largest: 1980-10-05 13:58:00
second_largest: 1980-10-05 13:57:00
(largest - second_largest):
Traceback (most recent call last):
File "", line 1, in
File "C:\Python27\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 540, in runfile
execfile(filename, namespace)
File "C:/Users/user1/Current projects//test/test_pyculiarity_0.py", line 17, in
print "(largest - second_largest): ", (largest - second_largest)
TypeError: unsupported operand type(s) for -: 'str' and 'str'
when i ran the "nosetests ." command, all 13 tests failed with the same TyperError.
any suggestions? Please advise, and i appreciate your time in advance. Thank you.
PS: sorry if my formatting, etc. is odd and not standard. this is my first issue posting. i can edit any clarifications you need.
The text was updated successfully, but these errors were encountered: