New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tzdata crash at startup in escript #86

Closed
sdanzan opened this Issue Oct 6, 2015 · 19 comments

Comments

9 participants
@sdanzan
Copy link

sdanzan commented Oct 6, 2015

I have an application packaged as a standalone executable (generated with mix escript.build) that uses Timex to parse dates.

When the application is run from the command line, it crashes in first call to Timex.DateFormat.parse with the "classic" stack trace:

10:45:31.085 [error] Task #PID<0.83.0> started from #PID<0.81.0> terminating
** (ArgumentError) argument error
(stdlib) :ets.lookup(:tzdata_current_release, :release_version)
(tzdata) lib/tzdata/release_reader.ex:41: Tzdata.ReleaseReader.current_release_from_table/0
(tzdata) lib/tzdata/release_reader.ex:13: Tzdata.ReleaseReader.simple_lookup/1
(tzdata) lib/tzdata/release_reader.ex:7: Tzdata.ReleaseReader.zone_and_link_list/0
(tzdata) lib/tzdata.ex:61: Tzdata.zone_exists?/1
lib/timezone/timezone.ex:91: Timex.Timezone.get/2
lib/parse/datetime/parser.ex:213: Timex.Parse.DateTime.Parser.update_date/4
lib/parse/datetime/parser.ex:121: Timex.Parse.DateTime.Parser.apply_directives/3

When I call the main function of my application directly from within iex -S mix with the same arguments, it does not crash. In both case the tzdata application is loaded and started (I checked by printing inspect(Application.loaded_applications) and inspect(Application.started_applications) at the beginning of my main function.

I use Timex 0.19.5 and tzdata 0.5.3.

@triptec

This comment has been minimized.

Copy link

triptec commented Oct 6, 2015

I've got the same error =(

@sdanzan

This comment has been minimized.

Copy link

sdanzan commented Oct 6, 2015

OK, escript builds do not support well data embeded in priv (which tzdata requires). However when trying to work around that I found a very strange behaviour: tzdata is present in the list of started applications, but when I call Application.app_dir(:tzdata, ...) I get an application not started error. This easily reproduceable with an application that only Enum.each all started applications and call Application.app_dir

Anyway, I guess to properly embed Timex/Tzdata in standalone executables, I´ll have to try some Code.prepend_path magic.

@bitwalker

This comment has been minimized.

Copy link
Owner

bitwalker commented Oct 6, 2015

@sdanzan Actually escripts do support reading data from their embedded archive (via erl_prim_loader). However the bigger problem is that tzdata is using ets:file2tab to load timezone information from an ETS dump, and the underlying implementation is using functionality which requires the file to be accessible directly, there doesn't appear to be a way to load the file as a binary and call a different function to read it.

@lau Is there a relatively straightforward fallback solution we can use with tzdata for these scenarios? One thought I had is that we could dump the ETS table to a list via tab2list, then write that to a file (as terms). We can then determine whether to load that file via erl_prim_loader or a simple File.read based on whether we can locate the tzdata priv directory. Then load the file and repopulate the table by iterating over the loaded list. It would be slower, but would work in both cases. Alternatively, we could dump both to a list and the binary (via tab2file), and choose the file we need based on the current execution environment. In either case, I think we definitely want tzdata to work for the escript use case. Thoughts?

@lau

This comment has been minimized.

Copy link

lau commented Oct 6, 2015

A feature that I am considering implementing soon is to be able to configure (using the elixir config system) an alternative dir location for the files that are currently in the priv dir. I don't know if it would be a practical solution for most people doing escripts, but I'm guessing it could solve the problem.

@Korbin73

This comment has been minimized.

Copy link

Korbin73 commented Oct 6, 2015

I'm also seeing this error but in a slightly different context. I'm not sure if this will help, but I have configured my .iex.exs file to have the following function that uses date bindings

call_service_with_cache = fn -> 
    params = {credentials, App.get_yesterdays_date_format, App.get_todays_date_format}
    garden_data = ApiCache.call_api(params, fn credentials, date_from, date_to -> 
        Api.get_garden_data(credentials, date_from, date_to)
    end)
end

I see this error when trying to format the date to a string.

@bitwalker

This comment has been minimized.

Copy link
Owner

bitwalker commented Oct 6, 2015

@lau Yeah the same problem will still exist for escript builds, since the only way to read files from the escript is via erl_prim_loader, and storing the files externally isn't practical for an escript since the idea is to have a portable executable. If tzdata can support loading the data into ETS via an alternative means other than :ets.file2tab, then that would be the ideal fallback for when we have to read from the escript archive. The problem as I mentioned is that supporting that case requires one more storage format for the ETS table contents (or just dumping the ETS file to a term and writing that to a file and using that as the sole storage format). I looked into what :ets.file2tab is doing under the covers to see if there is a way to operate on the raw binary content, but nothing is exposed, and it's using disk_log under the covers anyway which appears to require a file on disk which is accessible via normal file APIs. So we'd either have to emulate what it's doing and parse the ETS file manually (not hard, but not ideal), or provide two ways to load the ETS data (from an ETS file, or from a list of terms)

Since datetime parsing/formatting/etc. is probably not an uncommon scenario for some escript use cases, I think it's worth the time to figure out a good solution.

@bitwalker

This comment has been minimized.

Copy link
Owner

bitwalker commented Oct 6, 2015

@Korbin73 In your .iex.exs, add above that code snippet:

Application.load(:tzdata)
:ok = Application.ensure_started(:tzdata)

That should fix your particular issue.

@Korbin73

This comment has been minimized.

Copy link

Korbin73 commented Oct 6, 2015

Excellent! That worked. Thanks @bitwalker

@bitwalker bitwalker self-assigned this Oct 6, 2015

@lau

This comment has been minimized.

Copy link

lau commented Oct 7, 2015

@bitwalker I see. Having two different file formats might be messy, but going down that route might be a good idea. Maybe with the alternative file format as a separate package if possible.

Using tzdata ~> 0.1.8 would probably work as all the data are just included at compile time. Could you allow that in the timex mix.exs? Then people could specify ~> 0.1.8 in their mix.exs. Are escripts compiled before distribution? Otherwise the RAM requirements for compilation of 0.1.x could be an issue if you want to run the escript on e.g. a raspberry pi.

bitwalker added a commit that referenced this issue Oct 12, 2015

@bitwalker

This comment has been minimized.

Copy link
Owner

bitwalker commented Oct 12, 2015

@lau I've changed the requirements in timex to allow using tzdata 0.1.8 for now, until I can get around to my PR to Elixir's mix escript.build task, then I can tackle modifying tzdata and sending you a PR.

@sdanzan You can put the following in your mix.exs to use the above version of tzdata inside your escript instead of the current one:

def deps do
  [..., {:tzdata, "== 0.1.8", override: true}]
end

I'll update this thread when we have a longer term solution in place.

@bitwalker bitwalker changed the title tzdata crash at startup in standalone executable tzdata crash at startup in escript Oct 12, 2015

sdanzan added a commit to sdanzan/feedistiller that referenced this issue Oct 13, 2015

Switch to tzdata 0.1.8
This is currently required to get escript.build properly working
(see: bitwalker/timex#86)
@sdanzan

This comment has been minimized.

Copy link

sdanzan commented Oct 13, 2015

Yes, it works, thanks!

@markburns

This comment has been minimized.

Copy link

markburns commented Dec 11, 2015

I had this issue, but had a little trouble finding this github issue via google, so will add the stacktrace to make this easier to find for others

Could not start application tzdata: exited in: Tzdata.App.start(:normal, [])
    ** (EXIT) an exception was raised:
        ** (MatchError) no match of right hand side value: {:error, {:shutdown, {:failed_to_start_child, Tzdata.EtsHolder, {%ArgumentError{message: "unknown application: :tzdata"}, [{Application, :app_dir, 1, [file: 'lib/application.ex', line: 324]}, {Application, :app_dir, 2, [file: 'lib/application.ex', line: 333]}, {Tzdata.EtsHolder, :release_dir, 0, [file: 'lib/tzdata/ets_holder.ex', line: 86]}, {Tzdata.EtsHolder, :make_sure_a_release_dir_exists, 0, [file: 'lib/tzdata/ets_holder.ex', line: 70]}, {Tzdata.EtsHolder, :make_sure_a_release_is_on_file, 0, [file: 'lib/tzdata/ets_holder.ex', line: 64]}, {Tzdata.EtsHolder, :init, 1, [file: 'lib/tzdata/ets_holder.ex', line: 10]}, {:gen_server, :init_it, 6, [file: 'gen_server.erl', line: 328]}, {:proc_lib, :init_p_do_apply, 3, [file: 'proc_lib.erl', line: 240]}]}}}}
            (tzdata) lib/tzdata/tzdata_app.ex:15: Tzdata.App.start/2
            (kernel) application_master.erl:273: :application_master.start_it_old/4
@aborn

This comment has been minimized.

Copy link

aborn commented Jan 10, 2016

It works tzdata 0.1.8, Thanks @bitwalker

@lau

This comment has been minimized.

Copy link

lau commented Jan 10, 2016

I would use "~> 0.1.8" instead of "== 0.1.8". Because when the next version of the timezone data comes out, you would want 0.1.9 instead of 0.1.8. And ~> 0.1.8 allows "0.1.9", "0.1.10" and so on.

@bitwalker

This comment has been minimized.

Copy link
Owner

bitwalker commented Jan 11, 2016

Changed :)

@rauluranga

This comment has been minimized.

Copy link

rauluranga commented Jan 26, 2016

Hi to all!
I've got the same error, downgrading tzdata to 0.1.8 fix the build problem.

rauluranga added a commit to rauluranga/exjournal that referenced this issue Jan 26, 2016

added escript configuration
downgrade tzdata version [issue #86 bitwalker/timex#86]
@bitwalker

This comment has been minimized.

Copy link
Owner

bitwalker commented Mar 5, 2016

This is documented in 2.0, and I'm going to look in to a PR for a solution to fix this in more recent versions as soon as I can.

@bitwalker bitwalker closed this Mar 5, 2016

@mneumark

This comment has been minimized.

Copy link

mneumark commented Jan 12, 2017

@bitwalker Any word on a fix for this issue? Downgrading tzdata fixes the problem but its not the best solution in the world. Any reason the issue is closed if its not really resolved?

mneumark pushed a commit to rentpath/rprel that referenced this issue Jan 12, 2017

Micah Neumark
@bitwalker

This comment has been minimized.

Copy link
Owner

bitwalker commented Jan 13, 2017

The fundamental issue is that it's not possible to load ETS tables from files in an escript. The tzdata package relies on this functionality in all versions > 0.1.x. Basically, tzdata needs to be refactored such that it can operate under this constraint, and unfortunately I haven't had time to take a crack at a PR. It's closed because there is a workaround, even though it's not ideal, and it's really an issue that should be tracked against tzdata, not timex.

pasha added a commit to rentpath/rprel that referenced this issue Jan 13, 2017

[PL] [40] Create annotated tag before release
- added ability to create annotated tag prior to release
- updated build to use tar.gz as extension
- updated version and changelog
- updated timex to latest
- updated bypass
- updated Timex
- use `DateTime` instead of `Timex.Date`
- set logger to `:warn` on test
- added function to create tag
- added function to create tags ref
- fixed warnings
- display github error and fallback to defaults
- updated tests to reflect additional calls to fetch full commit sha
- fixed crash issue on compile by downgrading tzdata to 0.1.8
For more details, see:
bitwalker/timex#86

pasha added a commit to rentpath/rprel that referenced this issue Jan 13, 2017

[PL] [40] Create annotated tag before release
[Issue](#40)

- added ability to create annotated tag prior to release
- updated build to use tar.gz as extension
- updated version and changelog
- updated timex to latest
- updated bypass
- updated Timex
- use `DateTime` instead of `Timex.Date`
- set logger to `:warn` on test
- added function to create tag
- added function to create tags ref
- fixed warnings
- display github error and fallback to defaults
- updated tests to reflect additional calls to fetch full commit sha
- fixed crash issue on compile by downgrading tzdata to 0.1.8
For more details, see:
bitwalker/timex#86

pasha added a commit to rentpath/rprel that referenced this issue Jan 13, 2017

[PL] [40] Create annotated tag before release
[Issue](#40)

- added ability to create annotated tag prior to release
- updated build to use tar.gz as extension
- updated version and changelog
- updated timex to latest
- updated bypass
- updated Timex
- use `DateTime` instead of `Timex.Date`
- set logger to `:warn` on test
- added function to create tag
- added function to create tags ref
- fixed warnings
- display github error and fallback to defaults
- updated tests to reflect additional calls to fetch full commit sha
- fixed crash issue on compile by downgrading tzdata to 0.1.8
For more details, see:
bitwalker/timex#86

pasha added a commit to rentpath/rprel that referenced this issue Jan 13, 2017

[PL] [40] Create annotated tag before release
[Issue](#40)

- added ability to create annotated tag prior to release
- updated build to use tar.gz as extension
- updated version and changelog
- updated timex to latest
- updated bypass
- updated Timex
- use `DateTime` instead of `Timex.Date`
- set logger to `:warn` on test
- added function to create tag
- added function to create tags ref
- fixed warnings
- display github error and fallback to defaults
- updated tests to reflect additional calls to fetch full commit sha
- fixed crash issue on compile by downgrading tzdata to 0.1.8
For more details, see:
bitwalker/timex#86

pasha added a commit to rentpath/rprel that referenced this issue Jan 13, 2017

[PL] [40] Create annotated tag before release
[Issue](#40)

- added ability to create annotated tag prior to release
- updated build to use tar.gz as extension
- updated version and changelog
- updated timex to latest
- updated bypass
- updated Timex
- use `DateTime` instead of `Timex.Date`
- set logger to `:warn` on test
- added function to create tag
- added function to create tags ref
- fixed warnings
- display github error and fallback to defaults
- updated tests to reflect additional calls to fetch full commit sha
- fixed crash issue on compile by downgrading tzdata to 0.1.8
For more details, see:
bitwalker/timex#86

pasha added a commit to rentpath/rprel that referenced this issue Jan 13, 2017

[PL] [40] Create annotated tag before release
[Issue](#40)

- added ability to create annotated tag prior to release
- updated build to use tar.gz as extension
- updated version and changelog
- updated timex to latest
- updated bypass
- updated Timex
- use `DateTime` instead of `Timex.Date`
- set logger to `:warn` on test
- added function to create tag
- added function to create tags ref
- fixed warnings
- display github error and fallback to defaults
- updated tests to reflect additional calls to fetch full commit sha
- fixed crash issue on compile by downgrading tzdata to 0.1.8
For more details, see:
bitwalker/timex#86

pasha added a commit to rentpath/rprel that referenced this issue Jan 13, 2017

[PL] [40] Create annotated tag before release
[Issue](#40)

- added ability to create annotated tag prior to release
- updated build to use tar.gz as extension
- updated version and changelog
- created binary
- updated timex to latest
- updated bypass
- updated Timex
- use `DateTime` instead of `Timex.Date`
- set logger to `:warn` on test
- added function to create tag
- added function to create tags ref
- fixed warnings
- display github error and fallback to defaults
- updated tests to reflect additional calls to fetch full commit sha
- fixed crash issue on compile by downgrading tzdata to 0.1.8
For more details, see:
bitwalker/timex#86
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment