just started getting this IOError: write: broken pipe (EPIPE) #38

ghost · 2022-12-20T19:41:36Z

Just started happening since 12/16/22 did something change in Jib.jl or IB on or near 12/15/22 which is the last day the code ran.

┌ Warning: connection terminated
└ @ Jib.Reader ~/.julia/packages/Jib/JOmEj/src/reader.jl:70
ERROR: [ Info: reader exiting
LoadError: IOError: write: broken pipe (EPIPE)
Stacktrace:
 [1] uv_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64)
   @ Base ./stream.jl:1064
 [2] unsafe_write(s::Sockets.TCPSocket, p::Ptr{UInt8}, n::UInt64)
   @ Base ./stream.jl:1118
 [3] unsafe_write
   @ ./io.jl:683 [inlined]
 [4] write
   @ ./io.jl:706 [inlined]
 [5] write_one(socket::Sockets.TCPSocket, buf::IOBuffer)
   @ Jib.Client ~/.julia/packages/Jib/JOmEj/src/client.jl:28
 [6] sendmsg
   @ ~/.julia/packages/Jib/JOmEj/src/requests.jl:27 [inlined]
 [7] reqMktData(ib::Jib.Connection, tickerId::Int64, contract::Jib.Contract, genericTicks::String, snapshot::Bool, regulatorySnaphsot::Bool, mktDataOptions::NamedTuple{(), Tuple{}})
   @ Jib.Requests ~/.julia/packages/Jib/JOmEj/src/requests.jl:70
 [8] reqMktData
   @ ~/.julia/packages/Jib/JOmEj/src/requests.jl:47 [inlined]
 [9] top-level scope
   @ ~/tontine_2022/2022_live/9_6_22_LIVE_STK_IV_VOL_LAST.jl:171
in expression starting at /home/dave/tontine_2022/2022_live/9_6_22_LIVE_STK_IV_VOL_LAST.jl:160

running latest Jib.jl updated pkg and julia latest version.

@v1.8) pkg> up
    Updating registry at `~/.julia/registries/General.toml`
    Updating git-repo `https://github.com/lbilli/Jib.jl`
  No Changes to `~/.julia/environments/v1.8/Project.toml`
  No Changes to `~/.julia/environments/v1.8/Manifest.toml`

(@v1.8) pkg> up
    Updating registry at `~/.julia/registries/General.toml`
    Updating git-repo `https://github.com/lbilli/Jib.jl`
  No Changes to `~/.julia/environments/v1.8/Project.toml`
  No Changes to `~/.julia/environments/v1.8/Manifest.toml`

(@v1.8) pkg> st
Status `~/.julia/environments/v1.8/Project.toml`
  [c52e3926] Atom v0.12.38
  [336ed68f] CSV v0.10.8
  [1b08a953] Dash v1.1.2
⌃ [a93c6f00] DataFrames v1.3.6
  [1313f7d8] DataFramesMeta v0.12.0
  [31a5f54b] Debugger v0.7.6
⌅ [c43c736e] Genie v4.18.1
⌅ [cd3eb016] HTTP v0.9.17
  [f310f2d2] Jib v0.19.3 `https://github.com/lbilli/Jib.jl#master`
  [e5e0dc1b] Juno v0.8.4
  [f0f68f2c] PlotlyJS v0.18.10
⌃ [c3e4b0f8] Pluto v0.19.9
⌅ [4acbeb90] Stipple v0.24.5
⌃ [ec984513] StipplePlotly v0.12.4
⌃ [a3c5d34a] StippleUI v0.19.4
  [9e3dc215] TimeSeries v0.23.1
  [c2297ded] ZMQ v1.2.2
  [ade2ca70] Dates
  [56ddb016] Logging
  [10745b16] Statistics
Info Packages marked with ⌃ and ⌅ have new versions available, but those with ⌅ are restricted by compatibility constraints from upgrading. To see why use `status --outdated`

(@v1.8) pkg> 

dave@deepthought:~$ juliaup update
dave@deepthought:~$ juliaup status
 Default  Channel  Version                Update 
-------------------------------------------------
       *  release  1.8.3+0.x64.linux.gnu         
dave@deepthought:~$ juliaup update
dave@deepthought:~$ julia

this is the call that breaks

for (idx, s) in enumerate(eachrow(symbol_list))

    contract = Jib.Contract(symbol=s.Sym, secType="STK" , exchange=s.exchange , currency="USD")
    Jib.reqMktData(ib, idx, contract,"106", false) # https://interactivebrokers.github.io/tws-api/tick_types.html
end

BUT this one works properly. They both share the same ZMQ configuration code and worked fine up until 12/16/22.

for (idx, s) in enumerate(eachrow(symbol_list))
  sleep(5)
    contract = Jib.Contract(symbol=s.Sym, secType="STK" , exchange=s.exchange , currency="USD")
  try
       Jib.reqHistoricalData(ib, idx ,contract,"","1 Y","1 day","TRADES",true,1,false )
  
  catch e
           println("something went wrong with : " , s.Sym)
           stk_send_msg = "STK" *  "~" * s.Sy * "~" * "CLOSE" * "~" * string( 999 ) #show there was an error
           ZMQ.send( stk_socket, stk_send_msg )
           continue
  end
end

The text was updated successfully, but these errors were encountered:

lbilli · 2022-12-21T03:15:39Z

It looks you are getting disconnected by TWS, probably due to pacing issues.
TWS doesn't allow more then 50 requests per second, over any 5 second periods; i.e. 250 req / 5s.

You should check if you are getting any of these error messages:

  0 Warning: Approaching max rate of 50 messages per second (48)
 or
100 Max rate of messages per second has been exceeded:max=50

After sending error 100 TWS closes the connection immediately.

Workarounds:

carefully pace requests yourself not to exceed the threshold of 250 req / 5s
TWS before 10.16: the default behavior is to disconnect when threshold is breached.
Use connectOptions="+PACEAPI" when connecting and TWS will automatically pace the requests and avoid disconnections.
TWS 10.16 and later: the default behavior is to automatically pace requests unless option
API -> "Reject messages above maximum allowed message rate vs. applying pacing." is checked.

Ref: IB Release Notes

If running TWS pre 10.16 you might want to consider upgrading to a current version and keep the default behavior.

ghost · 2022-12-21T03:42:01Z

It looks you are getting disconnected by TWS, probably due to pacing issues. TWS doesn't allow more then 50 requests per second, over any 5 second periods; i.e. 250 req / 5s.

You should check if you are getting any of these error messages:
  0 Warning: Approaching max rate of 50 messages per second (48)
 or
100 Max rate of messages per second has been exceeded:max=50
After sending error 100 TWS closes the connection immediately.

Workarounds:
* carefully pace requests yourself not to exceed the threshold of 250 req / 5s

* TWS before 10.16: the default behavior is to disconnect when threshold is breached.
  Use `connectOptions="+PACEAPI"` when connecting and TWS will automatically pace the requests and avoid disconnections.

* TWS 10.16 and later: the default behavior is to automatically pace requests unless option
  API -> "Reject messages above maximum allowed message rate vs. applying pacing." is checked.
Ref: IB Release Notes

If running TWS pre 10.16 you might want to consider upgrading to a current version and keep the default behavior.

thank you so much for taking the time.

this code has been working for about 3 months without fault so this behavior started 12/16/22.

I checked the HELP for the TWS desktop I am using as a gateway and the build is 10.20.1f 12/13/22

I don't know how to put connectOptions="+PACEAPI" into my code.

ALSO I can't figure out how to check the log files. I "thought" that the log file was given in launcher.log but now all it says

2022-12-20 21:00:21.540 [SA] INFO  [JTS-PostAuthenticateS2-19] - performLogSwitch() [force=false]
2022-12-20 21:00:21.657 [SA] INFO  [JTS-PostAuthenticateS2-19] - Switching to a new encrypted log..

lbilli · 2022-12-21T10:00:55Z

I don't know how to put connectOptions="+PACEAPI" into my code.

That's an argument of Jib.connect(), as in:

ib = Jib.connect(4002, 1, "+PACEAPI")

Though, it shouldn't be necessary for TWS 10.20 if
Settings -> API -> "Reject messages above maximum allowed message rate vs. applying pacing." is unchecked.

You should check if you are getting any of these error messages:
 0 Warning: Approaching max rate of 50 messages per second (48)
or
100 Max rate of messages per second has been exceeded:max=50
After sending error 100 TWS closes the connection immediately.

ALSO I can't figure out how to check the log files. I "thought" that the log file was given in launcher.log but now all it says

These messages are not found in launcher.log. TWS sends them directly to your program, which should handle them properly via the callback:

wrap.error(id, errorCode, errorString, advancedOrderRejectJson)

Your wrap.error() should at least print or save the messages somewhere for diagnostic purposes.

ghost · 2022-12-21T11:53:28Z

I don't know how to put connectOptions="+PACEAPI" into my code.

That's an argument of Jib.connect(), as in:
ib = Jib.connect(4002, 1, "+PACEAPI")
Though, it shouldn't be necessary for TWS 10.20 if Settings -> API -> "Reject messages above maximum allowed message rate vs. applying pacing." is unchecked.
You should check if you are getting any of these error messages:
 0 Warning: Approaching max rate of 50 messages per second (48)
or
100 Max rate of messages per second has been exceeded:max=50
After sending error 100 TWS closes the connection immediately.
ALSO I can't figure out how to check the log files. I "thought" that the log file was given in launcher.log but now all it says

These messages are not found in launcher.log. TWS sends them directly to your program, which should handle them properly via the callback:
wrap.error(id, errorCode, errorString, advancedOrderRejectJson)
Your wrap.error() should at least print or save the messages somewhere for diagnostic purposes.

thank you for the quick reply.

I can confirm that Settings -> API -> "Reject messages above maximum allowed message rate vs. applying pacing." is unchecked.

I have the code from your EXCELLENT example (see below) is this enough to diagnose this issue please?
I could add logging to the code ( which I should have done in the first place).

wrap = Jib.Wrapper(
         # These events are always sent
         error= (id, errorCode, errorString, advancedOrderRejectJson) -> println("DJ Error: $(something(id, "NA")) $errorCode $errorString $advancedOrderRejectJson"),

REPLACED BY

wrap = Jib.Wrapper(
         # These events are always sent
         # error= (id, errorCode, errorString, advancedOrderRejectJson) -> println("DJ Error: $(something(id, "NA")) $errorCode $errorString $advancedOrderRejectJson"),
         error= (id, errorCode, errorString, advancedOrderRejectJson) ->  @info "message : $(now()) : DJ Error: $(something(id, "NA")) $errorCode $errorString $advancedOrderRejectJson"   ),

Got to say that it's weird this JUST started happening. I have a raspberry pi monitoring the ISP feed and there are no problems there.

lbilli · 2022-12-21T13:57:10Z

That is the wrap.error() I was referring to.

If the error messages I mentioned above don't show up in your logs then the root problem might be something different than sending too many requests too quickly.

If pacing is not the issue, browsing through your logs might give some clues of what is going on.

You can verify that your application indeed gets unexpectedly disconnected from TWS by checking:

isopen(ib.socket)  # = true when connected

or by clicking on DATA on the upper right corner of the TWS window.

ghost · 2022-12-22T15:30:17Z

problem just went away this morning BUT I am certainly going to go back and do this properly using your advice. Thank you for a wonderful development environment. I can't spend more than a hour a week on this BUT in that hour your code makes a HUGE difference. I learn by example and you REALLY do a great job. Thanks and happy hols to you and yours

ghost closed this as completed Dec 21, 2022

ghost reopened this Dec 22, 2022

ghost closed this as completed Dec 22, 2022

lbilli mentioned this issue Feb 6, 2023

Error in readBin(private$socket, raw(), n = len) : invalid 'n' argument lbilli/rib#23

Open

This issue was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

just started getting this IOError: write: broken pipe (EPIPE) #38

just started getting this IOError: write: broken pipe (EPIPE) #38

ghost commented Dec 20, 2022 •

edited by ghost

Loading

lbilli commented Dec 21, 2022

ghost commented Dec 21, 2022 •

edited by ghost

Loading

lbilli commented Dec 21, 2022

ghost commented Dec 21, 2022 •

edited by ghost

Loading

lbilli commented Dec 21, 2022

ghost commented Dec 22, 2022

just started getting this IOError: write: broken pipe (EPIPE) #38

just started getting this IOError: write: broken pipe (EPIPE) #38

Comments

ghost commented Dec 20, 2022 • edited by ghost Loading

lbilli commented Dec 21, 2022

ghost commented Dec 21, 2022 • edited by ghost Loading

lbilli commented Dec 21, 2022

ghost commented Dec 21, 2022 • edited by ghost Loading

lbilli commented Dec 21, 2022

ghost commented Dec 22, 2022

ghost commented Dec 20, 2022 •

edited by ghost

Loading

ghost commented Dec 21, 2022 •

edited by ghost

Loading

ghost commented Dec 21, 2022 •

edited by ghost

Loading