Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gc:arc - Illegal storage access #15954

Closed
SFR0815 opened this issue Nov 13, 2020 · 28 comments
Closed

gc:arc - Illegal storage access #15954

SFR0815 opened this issue Nov 13, 2020 · 28 comments

Comments

@SFR0815
Copy link

SFR0815 commented Nov 13, 2020

Hi!

First of all I have to apologize that I can not actually provide a simple code example causing the issue - the code I am running is a bit too large to be documented here. I have not yet been able to identify where this below mentioned behavior actually originates form.

What I observe is that "nim c -r some_file.nim" does not cause any issue

while "nim c -r --gc:arc some_file.nim" fails with

..\nim-1.4.0\lib\system\alloc.nim(955) alloc ..\nim-1.4.0\lib\system\alloc.nim(787) rawAlloc SIGSEGV: Illegal storage access. (Attempt to read from nil?)
with line of my code mentioned to fail just does some trivial stuff (like: result = result & "x")

@Araq
Copy link
Member

Araq commented Nov 13, 2020

with line of my code mentioned to fail just does some trivial stuff (like: result = result & "x")

Valid bug of course but you should really use result.add "x" instead, much faster and more idiomatic code.

@SFR0815
Copy link
Author

SFR0815 commented Nov 13, 2020

Thank's for the adivce - but any idea what the valid bug could be related to?

@Araq
Copy link
Member

Araq commented Nov 13, 2020

To this very statement (result = result & "x").

@timotheecour
Copy link
Member

@SFR0815 there are many things you can do to improve your bug report:

  • add echo type(result) and report it
  • show full stacktrace
  • does it happen without --gc:arc
  • check whether same root cause as --gc:arc SIGSEGV with ptr aliasing #13137
  • if result is cstring, echo result == nil
  • echo cast[int](result[0].addr) => this will help determine if address is from ROM or heap or stack
  • also run on devel
  • with above info in mind, minimize

without any minimum reproducible example, not much one can do about it

@SFR0815
Copy link
Author

SFR0815 commented Nov 19, 2020

I will go item by item. Order will be somewhat different:

No. 2: show full stacktrace

Traceback (most recent call last)
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\api\uiapi_requests\list\unit_test\ut_uiapi_list.nim(23) ut_uiapi_list
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\api\uiapi_requests\list\uiapi_list.nim(87) getApiBody_SubEntitySelectionList
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\api\uiapi_requests\list\uiapi_list.nim(42) getApiBody_NameList
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\api\uiapi_requests\list\name_list_api_body.nim(33) getApiBodyWithDbTableData
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\table_data\interfaces\db_data\specific_procs\reporting_date\reporting_period_selector.nim(430) getBusinessYear_OfDate
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\table_data\interfaces\db_data\specific_procs\reporting_date\reporting_period_selector.nim(344) getBusinessYears_OfClient
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\data_item_interface\i_data_item.nim(308) value=
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\date_time_with_tz_item.nim(44) setValue_DateTime_Impl
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\utils\iso8601.nim(398) normalizeDateTime
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\utils\iso8601.nim(394) getDateTime
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\utils\iso8601.nim(382) normalizeTimestamp
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\utils\iso8601.nim(369) getTimestamp
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\utils\iso8601.nim(287) getDate
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\utils\iso8601.nim(261) getDateOfDateString
C:\Users\steph\.choosenim\toolchains\nim-#devel\lib\system\alloc.nim(958) alloc0
C:\Users\steph\.choosenim\toolchains\nim-#devel\lib\system\alloc.nim(955) alloc
C:\Users\steph\.choosenim\toolchains\nim-#devel\lib\system\alloc.nim(787) rawAlloc
SIGSEGV: Illegal storage access. (Attempt to read from nil?)

@SFR0815
Copy link
Author

SFR0815 commented Nov 19, 2020

No. 3: does it happen without --gc:arc

No. Without --gc:arc everything is fine.

@timotheecour
Copy link
Member

timotheecour commented Nov 19, 2020

from the stacktrace + code in alloc.nim please also re-run your code with -d:useSysAssert and show stacktrace+errors

@SFR0815
Copy link
Author

SFR0815 commented Nov 19, 2020

nim c -r --gc:arc -d:useSysAssert ut_uiapi_list.nim

gives:

[SYSASSERT] rawAlloc: requested size too small

@timotheecour
Copy link
Member

even with --stacktrace:on ?

please put yourself in position of someone looking at your bug report; it's really hard to do anything about it without any context or minimized reproducible example.

what is the code of getDateOfDateString? or ideally, what's the content of iso8601.nim (eg put in some gist https://gist.github.com/)

@SFR0815
Copy link
Author

SFR0815 commented Nov 19, 2020

I fully understand that without a code example it's a mess. No clue on how I can help on this.

Please note the following:
This iso8601 module converts a variety of iso8601-type DateTime formats I am receiving form other sources into some standard that I am using throughout my code - it's procs are called in tons of occasions. I does under near-to-all circumstances work smoothly - even with --gc:arc. Thus, I have no clue on where to start isolating the issue ...

No. 1: add echo type(result) and report it

Just in case it is of help here comes the code of my proc that produces the issue:

Line 261 is " day = day & $character"
type of 'day' is string, see below

proc getDateOfDateString(date_string: string, spec_string: string): DateSpec =
  let spec_string = c_date_spec_string
  var year: string
  var month: string 
  var day: string 
  for char_position in countup(date_string.low, date_string.high):
    let character = date_string[char_position]
    let position_spec_characater = spec_string[char_position]
    case position_spec_characater
    of c_year_pointer:
      if not character.isDigit(): 
        raise newException(Iso8601Error,
          "Some part of year specification is not numeric. \n" &
          "   - non-numeric part   : " & $character & "\n" & 
          "   - on position        : " & $char_position & "\n" & 
          "   - in time zone string: " & date_string)
      year = year & $character
    of c_month_pointer:
      if not character.isDigit(): 
        raise newException(Iso8601Error,
          "Some part of month specification is not numeric. \n" &
          "   - non-numeric part   : " & $character & "\n" & 
          "   - on position        : " & $char_position & "\n" & 
          "   - in time zone string: " & date_string)
      month = month & $character
    of c_day_pointer:
      if not character.isDigit(): 
        raise newException(Iso8601Error,
          "Some part of day specification is not numeric. \n" &
          "   - non-numeric part   : " & $character & "\n" & 
          "   - on position        : " & $char_position & "\n" & 
          "   - in time zone string: " & date_string)
      #echo "day: " & day & " - character: " & $character
      day = day & $character
    of c_date_divisor: discard
    else: 
      raise newException(Iso8601Error, 
        "Irregular content specification on date specification: \n" &
        "   - character                 : " & $position_spec_characater & "\n" & 
        "   - on position               : " & $char_position & "\n" & 
        "   - in time zone specification: " & spec_string)
  if year.len() == 0: year = "1950"
  if month.len() == 0: month = "1"
  if day.len() == 0: day = "1"
  return DateSpec(
            year: year.parseInt().int16,
            month: month.parseInt().int16,
            day: day.parseInt().int16,
            date_string: date_string,
            spec_string: spec_string)
proc getDefaultDate(): DateSpec =
  return DateSpec(year: 1950, month: 1, day: 1)
proc getDate(date_string: string): DateSpec =
  let spec_string = c_date_spec_string
  if spec_string.len() != date_string.len():
    raise newException(Iso8601Error,
      "Date string provided is not compliant with ISO8601. \n" & 
      "     date string provided: " & date_string & " - of length " & $date_string.len() & "\n" & 
      "     spec string provided: " & spec_string & " - of length " & $spec_string.len() & "\n")
  return date_string.getDateOfDateString(spec_string)

I just introduce " echo "typedesc of day: " & $typedesc(day)" before line 261
... and run "nim c -r --gc:arc ut_uiapi_list.nim" again
... it prints tons of "typedesc of day: string"

... here comes the funny peace:
... usually it breaks at the same place BUT
... I saw it breaking somewhere else as well (in rare cases as it seems - seemingly somewhat difficult to reproduce):

Traceback (most recent call last)
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\api\uiapi_requests\list\unit_test\ut_uiapi_list.nim(33) ut_uiapi_list
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\api\uiapi_requests\list\uiapi_list.nim(87) getApiBody_SubEntitySelectionList
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\api\uiapi_requests\list\uiapi_list.nim(42) getApiBody_NameList
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\api\uiapi_requests\list\name_list_api_body.nim(21) getApiBodyWithDbTableData
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\api\uiapi_data_standards_2\request\source_db_data.nim(38) getDbData
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\api\uiapi_data_standards_2\request\source_db_data.nim(24) sourceByParentId
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\table_data\interfaces\db_data\base_type.nim(357) loadData_ByParentId
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\table_data\interfaces\db_data\base_type.nim(346) loadData_ByEntityId
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\table_data\interfaces\db_data\base_type.nim(317) loadData
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\table_data\interfaces\table_data\base_type.nim(274) loadData
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\db_read.nim(26) loadDataSets
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_statements\main.nim(86) getQueryJsonSql
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_statements\main.nim(50) getQuerySqlNode
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_statements\main.nim(29) getPreSelectFromNode
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_statements\main.nim(55) getQuerySqlNode
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_conditions\interfaces\column_condition\base_type.nim(400) getColumnConditions_SqlNode
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_conditions\interfaces\column_condition\base_type.nim(286) getLevelConditionBlock
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_conditions\interfaces\column_condition\base_type.nim(261) getLevelConditionPartitions
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_conditions\interfaces\column_condition\base_type.nim(73) getCopy
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_conditions\interfaces\column_condition\base_type.nim(70) getCopy
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_conditions\interfaces\column_condition\base_type.nim(65) getCopy
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_conditions\interfaces\column_condition\column_condition_value.nim(53) getCopy
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_conditions\interfaces\column_condition\column_condition_value.nim(49) getCopy
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\data_item.nim(108) getCopy
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\data_item_interface\i_data_item.nim(171) getValue_String
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\date_time_with_tz_item.nim(70) getValue_String_Impl
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\utils\iso8601.nim(387) getTimestamp
C:\Users\steph\.choosenim\toolchains\nim-#devel\lib\pure\times.nim(1578) initTimeFormat
C:\Users\steph\.choosenim\toolchains\nim-#devel\lib\system\strmantle.nim(283) nimCharToStr
C:\Users\steph\.choosenim\toolchains\nim-#devel\lib\system\alloc.nim(958) alloc0
C:\Users\steph\.choosenim\toolchains\nim-#devel\lib\system\alloc.nim(955) alloc
C:\Users\steph\.choosenim\toolchains\nim-#devel\lib\system\alloc.nim(787) rawAlloc
SIGSEGV: Illegal storage access. (Attempt to read from nil?)

Will try my luck on the gist

@SFR0815
Copy link
Author

SFR0815 commented Nov 19, 2020

@SFR0815
Copy link
Author

SFR0815 commented Nov 19, 2020

--stacktrace:on makes no difference

@SFR0815
Copy link
Author

SFR0815 commented Nov 19, 2020

No. 4: check whether same root cause as #13137
Could be. Quite a number of ref objects flying around.
BUT: (see comments above) Somewhat hard to believe that it is the code mentioned in the stacktrace ... (mentioned code usually works fine ...)

@SFR0815
Copy link
Author

SFR0815 commented Nov 19, 2020

No. 5: if result is cstring, echo result == nil
Variable is of string type.

@SFR0815
Copy link
Author

SFR0815 commented Nov 19, 2020

No. 6: echo castint => this will help determine if address is from ROM or heap or stack

Code is now:

    of c_day_pointer:
      if not character.isDigit(): 
        raise newException(Iso8601Error,
          "Some part of day specification is not numeric. \n" &
          "   - non-numeric part   : " & $character & "\n" & 
          "   - on position        : " & $char_position & "\n" & 
          "   - in time zone string: " & date_string)
      #echo "day: " & day & " - character: " & $character
      #echo "typedesc of day: " & $typedesc(day)
      echo "cast[int](day.addr): " & $cast[int](day.addr)
      if day.len() > 0: echo "    cast[int](day[0].addr): " & $cast[int](day[0].addr)
      day = day & $character

results in:

...
cast[int](day.addr): 15327120
cast[int](day.addr): 15327120
    cast[int](day[0].addr): 270089576
cast[int](day.addr): 15327424
cast[int](day.addr): 15327424
    cast[int](day[0].addr): 270089704
cast[int](day.addr): 15326704
cast[int](day.addr): 15326704
    cast[int](day[0].addr): 270090120
cast[int](day.addr): 15326704
cast[int](day.addr): 15326704
    cast[int](day[0].addr): 270089208
cast[int](day.addr): 15327136
cast[int](day.addr): 15327136
    cast[int](day[0].addr): 270089720
cast[int](day.addr): 15324048
cast[int](day.addr): 15324048
    cast[int](day[0].addr): 270089576
cast[int](day.addr): 15324240
cast[int](day.addr): 15324240
    cast[int](day[0].addr): 270089160
cast[int](day.addr): 15323328
cast[int](day.addr): 15323328
    cast[int](day[0].addr): 270089864
Traceback (most recent call last)
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\api\uiapi_requests\list\unit_test\ut_uiapi_list.nim(33) ut_uiapi_list
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\api\uiapi_requests\list\uiapi_list.nim(87) getApiBody_SubEntitySelectionList
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\api\uiapi_requests\list\uiapi_list.nim(42) getApiBody_NameList
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\api\uiapi_requests\list\name_list_api_body.nim(21) getApiBodyWithDbTableData
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\api\uiapi_data_standards_2\request\source_db_data.nim(38) getDbData
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\api\uiapi_data_standards_2\request\source_db_data.nim(24) sourceByParentId
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\table_data\interfaces\db_data\base_type.nim(357) loadData_ByParentId
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\table_data\interfaces\db_data\base_type.nim(346) loadData_ByEntityId
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\table_data\interfaces\db_data\base_type.nim(317) loadData
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\table_data\interfaces\table_data\base_type.nim(274) loadData
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\db_read.nim(26) loadDataSets
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_statements\main.nim(86) getQueryJsonSql
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_statements\main.nim(50) getQuerySqlNode
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_statements\main.nim(29) getPreSelectFromNode
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_statements\main.nim(55) getQuerySqlNode
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_conditions\interfaces\column_condition\base_type.nim(400) getColumnConditions_SqlNode
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_conditions\interfaces\column_condition\base_type.nim(286) getLevelConditionBlock
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_conditions\interfaces\column_condition\base_type.nim(261) getLevelConditionPartitions
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_conditions\interfaces\column_condition\base_type.nim(73) getCopy
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_conditions\interfaces\column_condition\base_type.nim(70) getCopy
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_conditions\interfaces\column_condition\base_type.nim(65) getCopy
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_conditions\interfaces\column_condition\column_condition_value.nim(53) getCopy
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_conditions\interfaces\column_condition\column_condition_value.nim(49) getCopy
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\data_item.nim(108) getCopy
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\data_item_interface\i_data_item.nim(171) getValue_String
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\date_time_with_tz_item.nim(70) getValue_String_Impl
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\utils\iso8601.nim(389) getTimestamp
C:\Users\steph\.choosenim\toolchains\nim-#devel\lib\pure\times.nim(1578) initTimeFormat
C:\Users\steph\.choosenim\toolchains\nim-#devel\lib\system\strmantle.nim(283) nimCharToStr
C:\Users\steph\.choosenim\toolchains\nim-#devel\lib\system\alloc.nim(958) alloc0
C:\Users\steph\.choosenim\toolchains\nim-#devel\lib\system\alloc.nim(955) alloc
C:\Users\steph\.choosenim\toolchains\nim-#devel\lib\system\alloc.nim(787) rawAlloc
SIGSEGV: Illegal storage access. (Attempt to read from nil?)

(was able to reproduce as mentioned above ...)

This just goes before the line causing the issue and provides:

@SFR0815
Copy link
Author

SFR0815 commented Nov 19, 2020

Comment:
lines 387 to 393:

proc getTimestamp*(date_time: times.DateTime): string =
  let full_date_time_format_string = getStandardDateTimeFormat_String() & "zzz"
  let full_date_time_format = full_date_time_format_string.initTimeFormat()
  result = date_time.format(full_date_time_format)
  if result.endsWith('Z'): 
    result.removeSuffix('Z')
    result = result & "+00:00"

@SFR0815
Copy link
Author

SFR0815 commented Nov 19, 2020

No. 7: also run on devel

Well, I was on devel (see above). Just updated to:

Nim Compiler Version 1.5.1 [Windows: amd64]
Compiled at 2020-11-19
Copyright (c) 2006-2020 by Andreas Rumpf

running nim c -r --gc:arc ut_uiapi_list.nim result in:

...
cast[int](day.addr): 15318128
cast[int](day.addr): 15318128
    cast[int](day[0].addr): 270679896
cast[int](day.addr): 15318128
cast[int](day.addr): 15318128
    cast[int](day[0].addr): 270679048
cast[int](day.addr): 15320240
cast[int](day.addr): 15320240
    cast[int](day[0].addr): 270679896
cast[int](day.addr): 15319872
cast[int](day.addr): 15319872
    cast[int](day[0].addr): 270679432
cast[int](day.addr): 15320688
cast[int](day.addr): 15320688
    cast[int](day[0].addr): 270679528
cast[int](day.addr): 15319968
cast[int](day.addr): 15319968
    cast[int](day[0].addr): 270679896
cast[int](day.addr): 15318128
cast[int](day.addr): 15318128
    cast[int](day[0].addr): 270679784
cast[int](day.addr): 15318128
cast[int](day.addr): 15318128
    cast[int](day[0].addr): 270679064
cast[int](day.addr): 15320240
cast[int](day.addr): 15320240
    cast[int](day[0].addr): 270679784
cast[int](day.addr): 15319872
cast[int](day.addr): 15319872
    cast[int](day[0].addr): 270678968
cast[int](day.addr): 15320688
cast[int](day.addr): 15320688
    cast[int](day[0].addr): 270679448
Traceback (most recent call last)
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\api\uiapi_requests\list\unit_test\ut_uiapi_list.nim(23) ut_uiapi_list
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\api\uiapi_requests\list\uiapi_list.nim(87) getApiBody_SubEntitySelectionList
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\api\uiapi_requests\list\uiapi_list.nim(42) getApiBody_NameList
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\api\uiapi_requests\list\name_list_api_body.nim(33) getApiBodyWithDbTableData
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\table_data\interfaces\db_data\specific_procs\reporting_date\reporting_period_selector.nim(430) getBusinessYear_OfDate
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\table_data\interfaces\db_data\specific_procs\reporting_date\reporting_period_selector.nim(333) getBusinessYears_OfClient
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\table_data\interfaces\db_data\specific_procs\reporting_date\reporting_period_selector.nim(257) getPeriods_OfClient
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\data_item_interface\i_data_item.nim(308) value=
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\date_time_with_tz_item.nim(44) setValue_DateTime_Impl
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\utils\iso8601.nim(401) normalizeDateTime
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\utils\iso8601.nim(397) getDateTime
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\utils\iso8601.nim(385) normalizeTimestamp
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\utils\iso8601.nim(372) getTimestamp
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\utils\iso8601.nim(290) getDate
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\utils\iso8601.nim(264) getDateOfDateString
C:\Users\steph\.choosenim\toolchains\nim-#devel\lib\system\alloc.nim(958) alloc0
C:\Users\steph\.choosenim\toolchains\nim-#devel\lib\system\alloc.nim(955) alloc
C:\Users\steph\.choosenim\toolchains\nim-#devel\lib\system\alloc.nim(787) rawAlloc
SIGSEGV: Illegal storage access. (Attempt to read from nil?)

@timotheecour
Copy link
Member

timotheecour commented Nov 19, 2020

@SFR0815 can you try to re-run with this modified code and show the relevant output?

if echo makes the bug go away there's other options (c_printf + friends)

proc getTimestamp*(date_time: times.DateTime): string =
  echo date_time
  let full_date_time_format_string = getStandardDateTimeFormat_String() & "zzz"
  echo full_date_time_format_string
  let full_date_time_format = full_date_time_format_string.initTimeFormat()
  echo full_date_time_format
  result = date_time.format(full_date_time_format)
  echo result
  if result.endsWith('Z'): 
    echo "ended_with_Z"
    result.removeSuffix('Z')
    echo result
    echo "before_crash"
    result = result & "+00:00"
    echo "after_crash"

@SFR0815
Copy link
Author

SFR0815 commented Nov 19, 2020

OK - will amending code accordingly on both occasions gives:

... 
Documenting getDateOfDateString: ######################################
before_crash
cast[int](day.addr): 15318128
after_crash

Documenting getDateOfDateString: ######################################
before_crash
cast[int](day.addr): 15318128
    cast[int](day[0].addr): 272383720
after_crash

Documenting getTimestamp: ######################################
2122-12-31T23:59:59-05:00
YYYY-MM-dd'T'HH:mm:ss'.'ffffffzzz
YYYY-MM-dd'T'HH:mm:ss'.'ffffffzzz
2122-12-31T23:59:59.000000-05:00

Documenting getDateOfDateString: ######################################
before_crash
cast[int](day.addr): 15318128
after_crash

Documenting getDateOfDateString: ######################################
before_crash
cast[int](day.addr): 15318128
    cast[int](day[0].addr): 272383000
after_crash

Documenting getTimestamp: ######################################
2021-01-01T00:00:00-05:00
YYYY-MM-dd'T'HH:mm:ss'.'ffffffzzz
YYYY-MM-dd'T'HH:mm:ss'.'ffffffzzz
2021-01-01T00:00:00.000000-05:00

Documenting getDateOfDateString: ######################################
before_crash
cast[int](day.addr): 15320240
after_crash

Documenting getDateOfDateString: ######################################
before_crash
cast[int](day.addr): 15320240
    cast[int](day[0].addr): 272383720
after_crash

Documenting getTimestamp: ######################################
2021-01-01T00:00:00-05:00
YYYY-MM-dd'T'HH:mm:ss'.'ffffffzzz
YYYY-MM-dd'T'HH:mm:ss'.'ffffffzzz
2021-01-01T00:00:00.000000-05:00

Documenting getDateOfDateString: ######################################
before_crash
cast[int](day.addr): 15319872
after_crash

Documenting getDateOfDateString: ######################################
before_crash
cast[int](day.addr): 15319872
    cast[int](day[0].addr): 272382904
after_crash

Documenting getTimestamp: ######################################
2020-12-31T00:00:00-05:00
YYYY-MM-dd'T'HH:mm:ss'.'ffffffzzz
YYYY-MM-dd'T'HH:mm:ss'.'ffffffzzz
2020-12-31T00:00:00.000000-05:00

Documenting getDateOfDateString: ######################################
before_crash
cast[int](day.addr): 15320688
after_crash

Documenting getDateOfDateString: ######################################
before_crash
cast[int](day.addr): 15320688
    cast[int](day[0].addr): 272383384
Traceback (most recent call last)
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\api\uiapi_requests\list\unit_test\ut_uiapi_list.nim(23) ut_uiapi_list
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\api\uiapi_requests\list\uiapi_list.nim(87) getApiBody_SubEntitySelectionList
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\api\uiapi_requests\list\uiapi_list.nim(42) getApiBody_NameList
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\api\uiapi_requests\list\name_list_api_body.nim(33) getApiBodyWithDbTableData
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\table_data\interfaces\db_data\specific_procs\reporting_date\reporting_period_selector.nim(430) getBusinessYear_OfDate
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\table_data\interfaces\db_data\specific_procs\reporting_date\reporting_period_selector.nim(333) getBusinessYears_OfClient
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\table_data\interfaces\db_data\specific_procs\reporting_date\reporting_period_selector.nim(257) getPeriods_OfClient
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\data_item_interface\i_data_item.nim(308) value=
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\date_time_with_tz_item.nim(44) setValue_DateTime_Impl
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\utils\iso8601.nim(413) normalizeDateTime
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\utils\iso8601.nim(409) getDateTime
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\utils\iso8601.nim(388) normalizeTimestamp
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\utils\iso8601.nim(375) getTimestamp
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\utils\iso8601.nim(293) getDate
D:\Documents\Programs\Projects\InvReporting\Prototype II\Progamming\invrep\src\db_model_2\db_table\data_set\item\implementations\utils\iso8601.nim(266) getDateOfDateString
C:\Users\steph\.choosenim\toolchains\nim-#devel\lib\system\alloc.nim(958) alloc0
C:\Users\steph\.choosenim\toolchains\nim-#devel\lib\system\alloc.nim(955) alloc
C:\Users\steph\.choosenim\toolchains\nim-#devel\lib\system\alloc.nim(787) rawAlloc
SIGSEGV: Illegal storage access. (Attempt to read from nil?)

@timotheecour
Copy link
Member

timotheecour commented Nov 19, 2020

thanks; from what I gathered it looks unlikely the bug is in getTimestamp, the gc corruption likely occured in some earlier place in your program
(especially based on ... I saw it breaking somewhere else as well (in rare cases as it seems - seemingly somewhat difficult to reproduce):), and that could be for any number of reason; without a full repro (even if not minimized) this will be a really hard exercise.

I was hoping -d:useSysAssert would give a stacktrace or lineinfo but didn't seem so from your snippet

if the code is not confidential, you could create a test branch + a cmdline to reproduce the bug; the more minimized it is the better for whoever looks at it

if it's a nim bug, it should be investigated and resolved though. What I usually do here when all else fails is painfully minimize:

create a test git branch in your code (to avoid messing with your normal branch in your code), then keep simplifying the code while retaining the bug; this is not a fun thing to do but it works.

hopefully one day nim catches up to D's dustmite (see https://github.com/CyberShadow/DustMite/wiki and #8276) and then this would all become automatable (dustmite is a joy to use in my experience)

@SFR0815
Copy link
Author

SFR0815 commented Nov 19, 2020

No. 8: with above info in mind, minimize & reproducable error etc.

Code is not confidential. Happy to share in whatever state. A big sorry for party messy WIP upfront. We do not keep it on GitHub but on our own server, though.

The issue occurs on an exercise that

  1. sources data from a Postgres-DB,
  2. loads it into a generic backend model that is then used to
  3. feed the data standard required by a web-api.
    Thus, not actually a piece of rocket science.

As I mentioned, --gc:arc worked smoothly on tons of tests on #1 & #2.

I would have been very surprised to see iso8601 being the cause of the issue. Thus, from my side, the outcome of today's matches expectations. Serves as a starting point.

In order to get closer to the issue, I would like to propose the following:
A) I will make an effort to locate the break a little better. Hopefully that provides some hint. Will not be this week - too busy otherwise. Will post outcomes as soon as I have some.
B) We will see where to go from there.

Cheers and thank you for your help
Stephan

@SFR0815
Copy link
Author

SFR0815 commented Nov 19, 2020

One last item re the above:

To be blunt, I am not actually from a planet in the vicinity of Betelgeuse, and, thus, my expertise on gc related issues is limited.

If I understand your comment "gc corruption likely ocured in some earlier place in your program" as:

There is some pointer associated to some ref object (addr) getting corrupted on the way and thus causes
the "SIGSEGV: Illegal storage access. (Attempt to read from nil?)" while trying to access.

Is that correct?

... It might be helpful to ensure a proper understanding of what I am searching for.

@timotheecour
Copy link
Member

timotheecour commented Nov 19, 2020

  • just a note: all your links (eg #1 etc) get interpreted by github as (clickable) cross-reference links pointing to this issue from those issues even though (eg #1) have nothing to do with this; so those links should be avoided, ideally edited to use some quotes (eg #1) ;-) thanks!

  • one more thing you can try on your side is running the program under a debugger (eg gdb/lldb) see also relevant options --stacktrace:on --debugger:native

Is that correct?

yes. That type of bug is harder to track down because the bug can occur earlier but the stacktrace only shows a symptom

  • if this is the place where it crashes result = result & "+00:00" you can also try to work around it by changing the code, eg:
result = result & "+00:00"
=>
result.add "+00:00" # not just workaround but actually better (but original bug still is a bug!)
# or even, avoiding touching `result` at all:
var ret: string
ret.add result
ret.add "+00:00"
return ret

@SFR0815
Copy link
Author

SFR0815 commented Nov 19, 2020

OK that serves as a starting point. Thus, will round up the usual suspects first.

@Araq
Copy link
Member

Araq commented Nov 19, 2020

Does your program run on Linux or OSX? If so, compile with --gc:arc -d:useMalloc -g and ask valgrind where it really crashes for the first time. :-)

@SFR0815
Copy link
Author

SFR0815 commented Nov 19, 2020

Unfortunately, it's Windows.

@SFR0815
Copy link
Author

SFR0815 commented Dec 9, 2020

Some update on the above:

I have made some changes on the famous iso8601 - a couple of improvements like using 'add' as suggested above.

This actually made the FIRST test of my test suite pass with success.

The SECOND test still ends with error (currently at some renderSQL command) - only difference to test no.1 is parameters

Errors raised are mainly related to string/char operations.

D:\Documents\Prototype II\Progamming\invrep\src\db_model_2\db_io\read\sql_conditions\implementations\invrep_column_condition.nim(31) getBasicCondition_SqlNode   
C:\Users\steph\.choosenim\toolchains\nim-1.4.2\lib\pure\parsesql.nim(1456) renderSQL    
C:\Users\steph\.choosenim\toolchains\nim-1.4.2\lib\pure\parsesql.nim(1244) ra    
C:\Users\steph\.choosenim\toolchains\nim-1.4.2\lib\pure\parsesql.nim(1197) add   
C:\Users\steph\.choosenim\toolchains\nim-1.4.2\lib\system\alloc.nim(958) alloc0 
C:\Users\steph\.choosenim\toolchains\nim-1.4.2\lib\system\alloc.nim(955) alloc
C:\Users\steph\.choosenim\toolchains\nim-1.4.2\lib\system\alloc.nim(787) rawAlloc

"thing" parameter on add in parsesql is a "="

@timotheecour
Copy link
Member

This actually made the FIRST test of my test suite pass with success.

good, but the underlying problem might still be there, correct? my understanding is that #16365 is a separate issue from this one; if so both need to be minimized (yes, it can take some work sometimes, but it always pays off)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants