Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change endtime - time >= 0 to endtime >= time #756

Merged
merged 1 commit into from Sep 11, 2023

Conversation

JYangQi00
Copy link
Collaborator

@JYangQi00 JYangQi00 commented Sep 11, 2023

What is the problem / what does the code in this PR do
strax._check_objects_non_negative_length checks to see if there are any data entries such that endtime-time >=0. However, when there is an error in the endtime such that endtime is very negative, endtime - time sees an overflow error, and native python would have endtime - time >0. Such an example can be seen in pema (see PR: #333) As such, native python would not detect such cases using _check_objects_non_negative_length. In actuality, it seems that numba has a check which somehow bypasses this overflow error, and detects such cases anyway but this is an instance of unexpected behavior.

Can you briefly describe how it works?
Simply change the statement np.all(strax.endtime(things) - things['time'] >=0) to np.all(strax.endtime(things) >= things['time']) .

Can you give a minimal working example (or illustrate with a figure)?

Please include the following if applicable:

  • Update the docstring(s)
  • Update the documentation
  • Tests to check the (new) code is working as desired.
  • Does it solve one of the open issues on github?

Please make sure that all automated tests have passed before asking for a review (you can save the PR as a draft otherwise).

@github-actions
Copy link

Coverage Status

coverage: 91.454%. remained the same when pulling adbe22a on non_negative_length_fix into a466a94 on master.

Copy link
Collaborator

@FaroutYLq FaroutYLq left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested and it is working. Just to give an example of what @JYangQi00 was talking about:

np.all(strax.endtime(things) >= things['time'])
False
np.all(strax.endtime(things) - things['time']>=0)
True

@FaroutYLq
Copy link
Collaborator

Unless @dachengx has objection, I would recommend merging. I don't think this will affect anything in regular data pipeline at all, but only affect those deep bugs (like wfsim) who would really have negative length. No worry for data.

@dachengx
Copy link
Collaborator

Tested and it is working. Just to give an example of what @JYangQi00 was talking about:

np.all(strax.endtime(things) >= things['time'])
False
np.all(strax.endtime(things) - things['time']>=0)
True

What are values in things here. for example?

@FaroutYLq
Copy link
Collaborator

Tested and it is working. Just to give an example of what @JYangQi00 was talking about:

np.all(strax.endtime(things) >= things['time'])
False
np.all(strax.endtime(things) - things['time']>=0)
True

What are values in things here. for example?

strax.endtime(things)
[ 1646832224892000512,  1646832224893727744,  1646832224942001152,
        1646832224943285760,  1646832224992000512,  1646832224992504576,
        1646832225042000896,  1646832225042932992,  1646832225092001024,
        1646832225092620544,  1646832225142001152,  1646832225144006656,
        1646832225192000512,  1646832225193914112,  1646832225242001152,
        1646832225243815680,  1646832225292000512,  1646832225292071424,
        1646832225342000896,  1646832225343009536,  1646832225392001024,
        1646832225393324544,  1646832225442001152,  1646832225443152384,
        1646832225492001024,  1646832225493915136,  1646832225542001152,
        1646832225543293952,  1646832225592000000,  1646832225593028096,
        1646832225642001152,  1646832225643642368,  1646832225692000512,
        1646832225692898304,  1646832225742001152,  1646832225743252992,
        1646832225792000768,  1646832225793705472,  1646832225842001152,
        1646832225843487488,  1646832225892000768,  1646832225892372736,
        1646832225942859264,  1646832225992001024,  1646832225992646144,
        1646832226042001152,  1646832226042044672,  1646832226092000512,
        1646832226093327872,  1646832226142000896,  1646832226143255808,
        1646832226192001024,  1646832226192347648,  1646832226242000640,
        1646832226243866624,  1646832226292000768,  1646832226293288192,
        1646832226342000896,  1646832226342102528,  1646832226392000256,
        1646832226392050432,  1646832226442000128,  1646832226442738688,
        1646832226492000512,  1646832226492409600,  1646832226542776320,
        1646832226592001024,  1646832226593710848,  1646832226642000128,
        1646832226642546944,  1646832226692001024,  1646832226693030400,
        1646832226742000896,  1646832226743301376,  1646832226792000256,
        1646832226792455680,  1646832226842000896,  1646832226843398144,
        1646832226892001024,  1646832226894094848,  1646832226942001152,
        1646832226944185344,  1646832226992000512,  1646832226992855552,
        1646832227042000384,  1646832227044162816,  1646832227092000256,
        1646832227092226560,  1646832227142001152,  1646832227144153856,
        1646832227192001024,  1646832227192057600,  1646832227242001152,
        1646832227243129088,  1646832227292001024,  1646832227292569856,
        1646832227342001152,  1646832227342477056,  1646832227392000256,
        1646832227393644032,  1646832227442000384,  1646832227442136064,
        1646832227492000512,  1646832227492027648,  1646832227542001152,
        1646832227543371264,  1646832227592001024,  1646832227592047616,
        1646832227642000896,  1646832227643204096,  1646832227692001024,
        1646832227693653248,  1646832227742001152,  1646832227743731712,
        1646832227792001024,  1646832227792308736,  1646832227842000128,
        1646832227843674368,  1646832227892001024,  1646832227893968128,
        1646832227942001152,  1646832227943568128,  1646832227992001024,
        1646832227994117888,  1646832228042001152,  1646832228043457280,
       -9223372036854775808,  1646832228092350464,  1646832228142000896,
        1646832228142564608,  1646832228192001024,  1646832228193793280,
        1646832228242000384,  1646832228243236864,  1646832228292000512,
        1646832228294109952,  1646832228342001152,  1646832228342465280,
        1646832228392001024,  1646832228394019584,  1646832228442000640,
        1646832228442770432,  1646832228492000768,  1646832228494025728,
        1646832228542000384,  1646832228543580672,  1646832228592001024,
        1646832228594107904,  1646832228642001152,  1646832228643683072,
        1646832228692001024,  1646832228693037824,  1646832228742001152,
        1646832228742900992,  1646832228792000512,  1646832228794195456,
        1646832228842001152,  1646832228842914304,  1646832228892001024,
        1646832228894036992,  1646832228942001152,  1646832228943844864,
        1646832228992001024,  1646832228992090624,  1646832229042001152,
        1646832229042949376,  1646832229092001024,  1646832229092128256,
        1646832229142001152,  1646832229143048192,  1646832229192001024,
        1646832229193540096, -9223372036854775808,  1646832229242611200,
        1646832229292000256,  1646832229293542144,  1646832229342001152,
        1646832229342101760])

things['time']
1646832224892000000, 1646832224893701120, 1646832224942000128,
       1646832224943258368, 1646832224992000000, 1646832224992490752,
       1646832225042000128, 1646832225042909696, 1646832225092000000,
       1646832225092602624, 1646832225142000128, 1646832225143965696,
       1646832225192000000, 1646832225193886208, 1646832225242000128,
       1646832225243785216, 1646832225292000000, 1646832225292065024,
       1646832225342000128, 1646832225342989056, 1646832225392000000,
       1646832225393301504, 1646832225442000128, 1646832225443130112,
       1646832225492000000, 1646832225493883648, 1646832225542000128,
       1646832225543273728, 1646832225592000000, 1646832225593008640,
       1646832225642000128, 1646832225643613952, 1646832225692000000,
       1646832225692881664, 1646832225742000128, 1646832225743229184,
       1646832225792000000, 1646832225793673728, 1646832225842000128,
       1646832225843460608, 1646832225892000000, 1646832225892359168,
       1646832225942847232, 1646832225992000000, 1646832225992628480,
       1646832226042000128, 1646832226042040320, 1646832226092000000,
       1646832226093302784, 1646832226142000128, 1646832226143235840,
       1646832226192000000, 1646832226192334336, 1646832226242000128,
       1646832226243828224, 1646832226292000000, 1646832226293265408,
       1646832226342000128, 1646832226342095872, 1646832226392000000,
       1646832226392046336, 1646832226442000128, 1646832226442717440,
       1646832226492000000, 1646832226492397568, 1646832226542764544,
       1646832226592000000, 1646832226593682688, 1646832226642000128,
       1646832226642531584, 1646832226692000000, 1646832226693006336,
       1646832226742000128, 1646832226743275008, 1646832226792000000,
       1646832226792442368, 1646832226842000128, 1646832226843371008,
       1646832226892000000, 1646832226894056960, 1646832226942000128,
       1646832226944151808, 1646832226992000000, 1646832226992840192,
       1646832227042000128, 1646832227044131584, 1646832227092000000,
       1646832227092218880, 1646832227142000128, 1646832227144123392,
       1646832227192000000, 1646832227192051712, 1646832227242000128,
       1646832227243105536, 1646832227292000000, 1646832227292554496,
       1646832227342000128, 1646832227342458368, 1646832227392000000,
       1646832227393624320, 1646832227442000128, 1646832227442131712,
       1646832227492000000, 1646832227492023296, 1646832227542000128,
       1646832227543346688, 1646832227592000000, 1646832227592043008,
       1646832227642000128, 1646832227643180544, 1646832227692000000,
       1646832227693620480, 1646832227742000128, 1646832227743708160,
       1646832227792000000, 1646832227792296960, 1646832227842000128,
       1646832227843649792, 1646832227892000000, 1646832227893939200,
       1646832227942000128, 1646832227943546880, 1646832227992000000,
       1646832227994088192, 1646832228042000128, 1646832228043431680,
       1646832228092000000, 1646832228092338688, 1646832228142000128,
       1646832228142557696, 1646832228192000000, 1646832228193760000,
       1646832228242000128, 1646832228243215360, 1646832228292000000,
       1646832228294082560, 1646832228342000128, 1646832228342448128,
       1646832228392000000, 1646832228393991168, 1646832228442000128,
       1646832228442752512, 1646832228492000000, 1646832228493994240,
       1646832228542000128, 1646832228543562240, 1646832228592000000,
       1646832228594073600, 1646832228642000128, 1646832228643653888,
       1646832228692000000, 1646832228693014016, 1646832228742000128,
       1646832228742875904, 1646832228792000000, 1646832228794168832,
       1646832228842000128, 1646832228842894080, 1646832228892000000,
       1646832228894013184, 1646832228942000128, 1646832228943815936,
       1646832228992000000, 1646832228992084736, 1646832229042000128,
       1646832229042928128, 1646832229092000000, 1646832229092120576,
       1646832229142000128, 1646832229143027712, 1646832229192000000,
       1646832229193512192, 1646832229242000000, 1646832229242595840,
       1646832229292000000, 1646832229293522176, 1646832229342000128,
       1646832229342094592])

You can see the negative endtime because of issues Jianyang described.

Copy link
Collaborator

@dachengx dachengx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks both!

@dachengx dachengx merged commit 1ca53c6 into master Sep 11, 2023
10 checks passed
@dachengx dachengx deleted the non_negative_length_fix branch September 11, 2023 18:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants