allow back-compat of timezone specifier

[jreback]

recently merged PR on fixing datetime w/tz parsing: #6453

we are hitting this in lots of places in here

I can simply put in a compat function to make this work with many versions of numpy (and not show the warning), but

it might make sense for back-compat to allow acceptance:

np.datetime64('2007-01-01 09:00:00.001Z') w/o showing the deprecation warning as this is de-facto (at present) a naive timezone in numpy anyhow AND this allows the same code to work in numpy < 1.11

(of course if a non-UTC tz, e.g. anything beyond the Z) then I would continue to do as the revised function.

[jreback]

cc @shoyer

[charris]

@shoyer This looks like the last thing needed for the 1.11.0 beta.

[charris]

@jreback Is your main concern a flood of deprecation warnings?

[jreback]

well you need different code in numpy < 1.11 from >= 1.11 to get equivalent results.

[charris]

So, perhaps a FutureWarning and a long wait?

[charris]

@jreback What would be the best fix from your point of view?

[jreback]

I @shoyer had the right soln. maybe leave the change along and if it is a UTC zone (e.g. a single 'Z') then show the FutureWarning but accept it. That way the user is notified but doesn't have to change it immediately (so its back compat).

[charris]

I think we need the back compat. What numpy versions do you officially support?

[jreback]

1.7+

[njsmith]

maybe leave the change along and if it is a UTC zone (e.g. a single 'Z') then show the FutureWarning but accept it. That way the user is notified but doesn't have to change it immediately (so its back compat).

That's exactly what current master & the 1.11 branch are doing, right? except that they use DeprecationWarning instead of FutureWarning? This is what deprecation warnings are for, to give a period where the old behavior continues to be accepted for back-compat :-)

[jreback]

@njsmith BUT the problem is that I cannot change it AND have compat with previous versions. So I have either to have to accept a warning (and catch it only in >= 1.11 and not change any code), or use a compat function so that I don't get a warning in prior versions.

I cannot simply change it if I want to maintain across numpy versions.

[charris]

Maybe PendingDeprecationWarning ;) Seeing how old 1.7 is, it looks like it will be a while... We do like to have backward compatible solutions available for deprecated features.

[njsmith]

@jreback: Still not sure I understand :-/. I think your options are:

1. Just ignore the warning and keep doing what you're doing; after a while bump your minimum required numpy version to 1.11 and switch to naive time strings.
2. Check the numpy version, and give old numpy the Zulu time strings and new numpy the naive time strings; eventually drop the old numpy compat code.

I don't really see what numpy can do that would give you any other options, other than (3) just don't make the change at all. But I think we do want to eventually get rid of allowing the timezone in the string for naive datetime64s -- if nothing else it's necessary to clear the way to having real timezone support later!

I can certainly see an argument that we should give the handling of Z timezones a longer deprecation period than we allow for other timezones, given that pandas uses them. Beyond that, what concretely do you want numpy to do?

[jreback]

guess I will have to do option 2.

[charris]

@jreback If you make a shim function, could you post it here? That way folks running into the warning might find it. Also, could the DeprecationWarning message be improved?

[jreback]

pandas-dev/pandas#12127

your messaging is good.

def tz_replacer(s):
    if isinstance(s, string_types):
        if s.endswith('Z'):
            s = s[:-1]
        elif s.endswith('-0000'):
            s = s[:-5]
    return s


def np_datetime64_compat(s, *args, **kwargs):
    """
    provide compat for construction of strings to numpy datetime64's with
    tz-changes in 1.11 that make '2015-01-01 09:00:00Z' show a deprecation
    warning, when need to pass '2015-01-01 09:00:00'
    """

    if not _np_version_under1p11:
        s = tz_replacer(s)
    return np.datetime64(s, *args, **kwargs)


def np_array_datetime64_compat(arr, *args, **kwargs):
    """
    provide compat for construction of an array of strings to a
    np.array(..., dtype=np.datetime64(..))
    tz-changes in 1.11 that make '2015-01-01 09:00:00Z' show a deprecation
    warning, when need to pass '2015-01-01 09:00:00'
    """

    if not _np_version_under1p11:

        # is_list_like
        if hasattr(arr, '__iter__') and not \
           isinstance(arr, string_and_binary_types):
            arr = [tz_replacer(s) for s in arr]
        else:
            arr = tz_replacer(s)

    return np.array(arr, *args, **kwargs)

[jakirkham]

Sorry to be pushy, but this is the last thing on the 1.11.0 release milestone. Is there something that is actionable here? I am willing to help if there is some clear actionable. Alternatively, could we make an alpha release while anything else here gets resolved and then add it in a beta?

[jreback]

I guess nothing to do

[charris]

Closing this then.