Unclear which timezone between_time uses

[Pirat83]

Hi @edtechre,

I have currently some issues with timezones. Especially between_time has unexpected behavior. So normally I integrate all my software systems in UTC timezone, since it is everywhere the same and there are no issues with daylight saving time and so on.

When integrating trading systems with other systems it can be also an option to use the US /Eastern timezone. So I have taken a look at pybroker.

Here is what happens (in my opinion):
I can assrue that start_date and end_date are set up to use the US/Eastern timezone when I create them.

• When creating Alpacas StockBarsRequest https://github.com/edtechre/pybroker/blob/master/src/pybroker/data.py#L357 both dates are converted to UTC which is fine
  

• After the data is fetched from Alpaca the DataFrame is in UTC which is fine:
  

• https://github.com/edtechre/pybroker/blob/master/src/pybroker/data.py#L386 converts the DataFrame to US/Estern time which is not ideal for me but it works good enough.
  

• Then the data is fiterded: https://github.com/edtechre/pybroker/blob/master/src/pybroker/strategy.py#L1179

• After executing: https://github.com/edtechre/pybroker/blob/master/src/pybroker/strategy.py#L1346 the timezone is None and the between_time filtering is applied so I wanted to filter it by (9:30,16:00) US/Estern.
  

• But since the date column in the DataFrame does not have any timezone any more my DataFrame is broken after this method:
  

So I will try to filter it with time in UTC timezone or None (which is supprisingly my local timezone)

I would strongly encourage using time with timezone instate of string tuples in between_time. Maybe this thing can be done easier if between_time is a tuple of time objects?

[edtechre]

Hi @Pirat83,

Thank you for pointing this out. Yes, I can add an option that allows between_time to be a tuple of time objects so that timezone information is preserved.

[edtechre]

Hi @Pirat83,

Sorry for the delay. I am looking at this now.

I am not able to reproduce the issue you are having. Could you provide sample code here? Also, which OS and version of Pandas are you using? I noticed that timezone conversion seems to work differently on Windows in Pandas.

FWIW, the between_time parameter ends up using Panda's between_time method to filter dates.

[Pirat83]

Hi @edtechre, I am using Linux and Pandas < 2.0.0. So I think it is the latest 1.x.y version.

[edtechre]

Thank you @Pirat83. That is the same as my setup. Can you provide me with sample code?

[Pirat83]

Hi @edtechre, Yes of course.

[edtechre]

Hi @Pirat83,

Let me know if this is still an issue.

[Pirat83]

Oh sorry. I will provide the code this week or close the ticket.

[Pirat83]

Hi @edtechre - I have added some example code to https://github.com/Pirat83/pybroker-experiments/tree/issue-69-and-51

[Pirat83]

So one of the main challenges is that pybroker dropps the timezone and then continues working with datetime without timezones.

datetimes are still in the US/Eastern timezone even if the timezone is not present any more. If someone want's to filter for some candles then ´datetime´s need to be used without timezone but they must match the US/Eastern timezone.

Since:

        result: TestResult = self.backtest(
            self.start_date_time, self.end_date_time, self.time_frame, ('9:30', '16:00'),
            [Day.MON, Day.TUES, Day.WEDS, Day.THURS, Day.FRI]
        )

can not be used if start_date_time and end_date_time additional filtering needs to be introduded in _weitght and _execute like this:

        result: TestResult = self.backtest(
            timeframe=self.time_frame, between_time=('9:30', '16:00'),
            days=[Day.MON, Day.TUES, Day.WEDS, Day.THURS, Day.FRI]
        )

Please correct me if I am wrong. Thank you very much or did not understand somehting correctly.

Thank you very much for your time and work.

[edtechre]

Hi @Pirat83,

Sorry for the late response. The timezone information is dropped in order to support filtering backtest data by the start/end date arguments, because those are datetime objects without timezone information. This happens independently of filtering with between_time.

Note, filtering with between_time uses Pandas' between_time filtering.

I would suggest performing any filtering on your DataFrame before executing your backtest on it. This can be done by querying your DataSource for the DataFrame, and then passing the modified DataFrame as your DataSource to the Strategy. See this notebook for an example.

If that is not enough, then I suggest assuming that you are dealing with times that have been normalized to US/Eastern, as you mentioned.