Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upcoming changes in time series tables #1250

Open
CSSEGISandData opened this issue Mar 22, 2020 · 196 comments
Open

Upcoming changes in time series tables #1250

CSSEGISandData opened this issue Mar 22, 2020 · 196 comments

Comments

@CSSEGISandData
Copy link
Owner

@CSSEGISandData CSSEGISandData commented Mar 22, 2020

We will update the time series tables in the following days, aiming to provide a cleaner and more organized dataset consistent with our new/current naming convention. We will also be reporting a new variable (i.e, testing), as well as data at the county level for the US. All files will continue to be updated daily around 11:59PM UTC.

The followiing specific changes will be made:

  • Three new time series tables will be added for the US. The first two will be the confirmed cases and deaths, reported at the county level. The third, number of tests conducted, will be reported at the state level. These new tables will be named time_series_covid19_confirmed_US.csv, time_series_covid19_deaths_US.csv, time_series_covid19_testing_US.csv, respectively.

  • Changes to the current time series include the removal of the US state and county-level entries, which will be replaced with a new single country level entry for the US. The tables will be renamed time_series_covid19_confirmed_global.csv and time_series_covid19_deaths_global.csv, and time_series_covid19_testing_global.csv, respectively.

  • The ISO code will be added in the global time series tables.

  • The FIPS code will be added in the new US time series tables.

  • We will no longer provide recovered cases.

  • The current set of time series files will be moved to our archive folder, and the new files will be added to the current folder.

Thanks!

@DataChant

This comment has been minimized.

Copy link

@DataChant DataChant commented Mar 22, 2020

Will recovered cases still be reported on the daily CSV files? Will they reflect the daily recovered or aggregated?

@CSSEGISandData

This comment has been minimized.

Copy link
Owner Author

@CSSEGISandData CSSEGISandData commented Mar 22, 2020

@DataChant No recovered cases will be reported in the daily reports and the time series tables.

@paolinic03

This comment has been minimized.

Copy link

@paolinic03 paolinic03 commented Mar 22, 2020

Woah, major news. Let’s do this. Bummed about no recovered but seems to be difficult to collect. County level data is going to be massive. Thank you

@billyburgoa

This comment has been minimized.

Copy link

@billyburgoa billyburgoa commented Mar 22, 2020

Thanks for your work. I'd like to know why you won't report or provide recovered cases.

@CSSEGISandData

This comment has been minimized.

Copy link
Owner Author

@CSSEGISandData CSSEGISandData commented Mar 22, 2020

No reliable data source reporting recovered cases for many countries, such as the US.

@ryanwoconnor

This comment has been minimized.

Copy link

@ryanwoconnor ryanwoconnor commented Mar 22, 2020

Can you please provide us a date/time for that cutover?
Can we place these new files into a different folder and leave the old files in place?
This way current dashboards that we may have running won't be full of errors when the cutover happens?

Thank you,
Ryan

@DMiradakis

This comment has been minimized.

Copy link

@DMiradakis DMiradakis commented Mar 22, 2020

Thanks so much! I'm making a Power BI Report now, so it's good to know about these upcoming changes!

@bevanward

This comment has been minimized.

Copy link

@bevanward bevanward commented Mar 22, 2020

Thanks @CSSEGISandData
With respect to your second bullet point, will Province/State remain for countries (excluding US) where you can source the data?

Changes look good - thanks for all the hard work - this is a very important data set!
Bevan

@christophGeoHealthCentre

This comment has been minimized.

Copy link

@christophGeoHealthCentre christophGeoHealthCentre commented Mar 22, 2020

How do you count actice cases without having recovered available?

@paolinic03

This comment has been minimized.

Copy link

@paolinic03 paolinic03 commented Mar 22, 2020

You don’t, just confirmed, deaths, and testing.

@DMiradakis

This comment has been minimized.

Copy link

@DMiradakis DMiradakis commented Mar 22, 2020

How do you count actice cases without having recovered available?

I'm just grouping the difference together into a group called "Active or Recovered". Like @paolinic03 said , it's the best we can do for the moment.

@analyzewithpower

This comment has been minimized.

Copy link

@analyzewithpower analyzewithpower commented Mar 22, 2020

THANK YOU!!! :)

@shahesam84

This comment has been minimized.

Copy link

@shahesam84 shahesam84 commented Mar 23, 2020

Will there be a release for those mentioned tables today? I don't see US tables yet.

@aatishb

This comment has been minimized.

Copy link

@aatishb aatishb commented Mar 23, 2020

Thank you for this. This really is an amazing resource, and I'm excited for these changes. I recommend pinning this issue so that folks don't miss it. https://help.github.com/en/github/managing-your-work-on-github/pinning-an-issue-to-your-repository

@hlianggithub

This comment has been minimized.

Copy link

@hlianggithub hlianggithub commented Mar 28, 2020

Thank you so much for this data resource. But could you still provide the data for each state of the US, which are also very helpful, useful, and critical for local people.

@ajfarkas

This comment has been minimized.

Copy link

@ajfarkas ajfarkas commented Mar 28, 2020

Is there some shorthand for unscientific dashboards to estimate recoveries? In a situation where no one dies, I could imagine that the number of recovered each day equals the increase in cases two weeks ago.
Would it be a good estimate to do that, but subtract any deaths that day (acknowledging this is a very rough estimate, since that death could be from any other day's cohort)?

@jp-fosterson

This comment has been minimized.

Copy link

@jp-fosterson jp-fosterson commented Mar 28, 2020

The New York Times is now compiling state and county data.

@web-sys1

This comment has been minimized.

Copy link

@web-sys1 web-sys1 commented Mar 28, 2020

@ArikVas

This comment has been minimized.

Copy link

@ArikVas ArikVas commented Mar 28, 2020

No reliable data source reporting recovered cases for many countries, such as the US.

I noticed the file time_series_covid19_recovered_global in the CSSE daily-timeseries repo; it was updated four hours ago. Does this mean that the reliability of the recovered data has improved (and if so, how/why)?

There are also less countries in recovered, what is the difference?

@nosamttam

This comment has been minimized.

Copy link

@nosamttam nosamttam commented Mar 28, 2020

Why did you remove the Province/State data?

@chadfurman

This comment has been minimized.

@jcampos8782

This comment has been minimized.

Copy link

@jcampos8782 jcampos8782 commented Mar 28, 2020

Why did you remove the Province/State data?

https://github.com/CSSEGISandData/COVID-19/blob/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv

It's clearly still there

I think @nosamttam is referring to United States state data. Not defending him since the whole point of this issue is to point out that its being moved to a separate set of files, but at least I think thats what he meant.

@johnorjias

This comment has been minimized.

Copy link

@johnorjias johnorjias commented Mar 28, 2020

@tmwmott

This comment has been minimized.

Copy link

@tmwmott tmwmott commented Mar 28, 2020

Please make known when the new file containing US time series data will be available. 6 days ago you told us files would change. When will the announced changes be complete?

@plorch

This comment has been minimized.

Copy link

@plorch plorch commented Mar 29, 2020

@tmwmott while you are waiting, you could start using a source like the one I list below. I switched to it and am summing across counties by state from the covid_confirmed_usafacts.csv, for example.
https://static.usafacts.org/public/data/covid-19/covid_confirmed_usafacts.csv', and https://static.usafacts.org/public/data/covid-19/covid_deaths_usafacts.csv

You could also switch to https://github.com/nytimes/covid-19-data/blob/master/us-states.csv

@tautme

This comment has been minimized.

Copy link

@tautme tautme commented Mar 29, 2020

@meldehi

This comment has been minimized.

Copy link

@meldehi meldehi commented Mar 29, 2020

Hey @CSSEGISandData! I know a lot of people are not appreciating what you are doing and the level of criticism is very unhealthy here but do not give up on the community.

@hrmartins

This comment has been minimized.

Copy link

@hrmartins hrmartins commented Mar 29, 2020

Hi,

My first word is thanks @CSSEGISandData for the info and hard work.

I just need a clarification. 5 days ago I had to remove all the info from our reports regarding the recovered cases. Only the confirmed and death files were available.

It’s important to my company to have reliable and trustable sources, so I removed this info instead of change the source.

However as I can see the recovered file is back again since 3-4 days and updated.

So please... I don’t need a bunch of different comments about this. I just need an official answer on what to do or what will happen.

Can I restore the old reports with the recovered info, or not?

Best regards and keep the good work.

@yurisong1

This comment has been minimized.

Copy link

@yurisong1 yurisong1 commented Mar 29, 2020

  • Three new time series tables will be added for the US. The first two will be the confirmed cases and deaths, reported at the county level. The third, number of tests conducted, will be reported at the state level. These new tables will be named time_series_covid19_confirmed_US.csv, time_series_covid19_deaths_US.csv, time_series_covid19_testing_US.csv, respectively.

testing data are so critical for analysis. Can you let us know when this update happens? Thanks in advance.

@jen2014

This comment has been minimized.

Copy link

@jen2014 jen2014 commented Mar 29, 2020

Thanks to @plorch for the links
https://static.usafacts.org/public/data/covid-19/covid_confirmed_usafacts.csv and https://static.usafacts.org/public/data/covid-19/covid_deaths_usafacts.csv, and thanks to those collecting the data. These give a very valuable and detailed view of the US situation.
For some reason, the first one ('confirmed) has not been updated for yesterday (March 28)

@jen2014

This comment has been minimized.

Copy link

@jen2014 jen2014 commented Mar 29, 2020

Just found out the series:
https://static.usafacts.org/public/data/covid-19/covid_confirmed_usafacts.csv
has been updated to March 28 in the last hour.

@plorch

This comment has been minimized.

Copy link

@plorch plorch commented Mar 29, 2020

@jen2014 AFAIK this is scraped from web pages. There is often an extra column "X" with no data and occaisionally a comma sneaks in, making my attempts to sum fail, but I have workarounds.

@jen2014

This comment has been minimized.

Copy link

@jen2014 jen2014 commented Mar 29, 2020

@plorch So far it works for me, with some minor cleaning up. I use a Ruby script to get the numbers, sort and sum them up.

@plorch

This comment has been minimized.

Copy link

@plorch plorch commented Mar 29, 2020

@jen2014 I am doing the same in R.

@alainivars

This comment has been minimized.

Copy link

@alainivars alainivars commented Mar 29, 2020

No reliable data source reporting recovered cases for many countries, such as the US.

Fist Thanks for your work,
but no reliable source! https://www.worldometers.info/coronavirus/ is one of your source and it provide it.
By the way the number of recovered permit to retrieve the number of active case and without this information the only way that this epidemic could end is the dead of everybody!

@bennett000 bennett000 mentioned this issue Mar 29, 2020
@ephsmith

This comment has been minimized.

Copy link

@ephsmith ephsmith commented Mar 29, 2020

@CSSEGISandData, 🙏🏻thank you🙏🏻 for your dashboards and the contributions you have made here.

The lack of focus on the actual post and the number of complaints are unsettling and counter to the level of professionalism we should all aspire to.

It’s worth noting that the owner’s actual application data is being refreshed and is currently powering their own dashboard.

The data provided here is a gift/community service. Let’s be thankful for what’s available and not complain that someone else is no longer doing work for our own dashboards etc.

@CSSEGISandData, keep up the work. We hope to see you active here again soon.

@BJReplay

This comment has been minimized.

Copy link

@BJReplay BJReplay commented Mar 29, 2020

Hope this helps anyone using this data in non-US regions: Power Query to combine all three (confirmed, deaths, recovered) into a single result set.

Should also work in US (but I haven't tried it). If it doesn't, just remove , "en-US" from each of the TransformColumnTypes

let
    Deaths = Csv.Document(Web.Contents("https://github.com/CSSEGISandData/COVID-19/raw/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_deaths_global.csv"),[Delimiter=",", Columns=71, Encoding=65001, QuoteStyle=QuoteStyle.None]),
    DeathsHeaders = Table.PromoteHeaders(Deaths, [PromoteAllScalars=true]),
    DeathsUnPivot = Table.UnpivotOtherColumns(DeathsHeaders, {"Province/State", "Country/Region", "Lat", "Long"}, "Date", "Deaths"),
    DeathsChangeTypes = Table.TransformColumnTypes(DeathsUnPivot,{{"Lat", type number}, {"Long", type number}, {"Date", type date}, {"Deaths", Int64.Type}}, "en-US"),
    Confirmed = Csv.Document(Web.Contents("https://github.com/CSSEGISandData/COVID-19/raw/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv"),[Delimiter=",", Columns=71, Encoding=65001, QuoteStyle=QuoteStyle.None]),
    ConfirmedHeaders = Table.PromoteHeaders(Confirmed, [PromoteAllScalars=true]),
    ConfirmedUnPivot = Table.UnpivotOtherColumns(ConfirmedHeaders, {"Province/State", "Country/Region", "Lat", "Long"}, "Date", "Confirmed"),
    ConfirmedChangeTypes = Table.TransformColumnTypes(ConfirmedUnPivot,{{"Lat", type number}, {"Long", type number}, {"Date", type date}, {"Confirmed", Int64.Type}}, "en-US"),
    Recovered = Csv.Document(Web.Contents("https://github.com/CSSEGISandData/COVID-19/raw/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_recovered_global.csv"),[Delimiter=",", Columns=71, Encoding=65001, QuoteStyle=QuoteStyle.None]),
    RecoveredHeaders = Table.PromoteHeaders(Recovered, [PromoteAllScalars=true]),
    RecoveredUnPivot = Table.UnpivotOtherColumns(RecoveredHeaders, {"Province/State", "Country/Region", "Lat", "Long"}, "Date", "Recovered"),
    RecoveredChangeTypes = Table.TransformColumnTypes(RecoveredUnPivot,{{"Lat", type number}, {"Long", type number}, {"Date", type date}, {"Recovered", Int64.Type}}, "en-US"),
    ConfirmedRecovered = Table.Join(ConfirmedChangeTypes,{"Country/Region", "Province/State", "Lat", "Long", "Date"},RecoveredChangeTypes,{"Country/Region", "Province/State", "Lat", "Long", "Date"}),
    ConfirmedRecoveredDeaths = Table.Join(ConfirmedRecovered,{"Country/Region", "Province/State", "Lat", "Long", "Date"},DeathsChangeTypes,{"Country/Region", "Province/State", "Lat", "Long", "Date"})
in
    ConfirmedRecoveredDeaths
@sibblegp

This comment has been minimized.

Copy link

@sibblegp sibblegp commented Mar 30, 2020

Three new time series tables will be added for the US. The first two will be the confirmed cases and deaths, reported at the county level. The third, number of tests conducted, will be reported at the state level. These new tables will be named time_series_covid19_confirmed_US.csv, time_series_covid19_deaths_US.csv, time_series_covid19_testing_US.csv, respectively.

When is this happening?????

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.