This project is a web scraper for https://engaginggospeldoctrine.org in order to take the correct URLs from the website for older podcast episodes. Everything prior to 1/18/2018's episode: 273.3: The Fall of Humanity (OT Gospel Doctrine Lesson 4, Study Notes) doesn't show up on the official podcast feed (as of 6/30/2019). The workaround presented in this code is to scrape the current contents of the whole site (currently it gets everything available, not just older episodes), save important information as a JSON file, then parse the JSON file into a readable podcast feed. Here's the breakdown of how to run the code.
# installs python3 dependencies in the Pipfile
pipenv install
pipenv shell
# scrapes the full website, with default settings to autothrottle the traffic
scrapy crawl egd -o output.json
# generates/overwrites the podcast.xml podcast feed
python main.py
The end result is that you can point your podcast app to this URL and it should work for all the older episodes! https://raw.githubusercontent.com/flamableconcrete/egd-rss-creator/master/feed.xml
- 07/04/2012 002: Book of Mormon Lesson 25: Alma 17-22 - audio for episode 25. Should be
2012/07/egd-bom-025.mp3 - 07/04/2012 003: Book of Mormon Lesson 26: Alma 23-29 - audio for episode 26. Should be
2012/07/egd-bom-026.mp3 - 06/05/2016 195: Introspection and Atonement (BoM Gospel Doctrine Lesson 22) - audio for episode 22. Should be
2019/02/egd-bom-022.mp3
These episodes have no audio on their pages. All but the first and last are from 2014 (Old Testament)
- 03/01/2013 036: Doctrine & Covenants Lesson 10: This Is My Voice unto All - Should be
2019/02/egd-dcch-010.mp3 - 078a - 105.1
- 01/02/2014 078a: This is My Work and My Glory; OT lesson 1(Full)
- 01/02/2014 078b: This is My Work and My Glory; OT lesson 1(Core)
- 01/11/2014 079a: Pre-Existence and Book of Abraham; OT Lesson 2 (Full)
- 01/11/2014 079b: Pre-Existence and Book of Abraham; OT Lesson 2 (Core)
- 01/17/2014 080.1: The Creation; OT Lesson 3 (Core)
- 01/17/2014 080.2: The Creation; OT Lesson 3 (Study Notes)
- 01/22/2014 081.1: The Fall; OT Lesson 4 (Core)
- 01/22/2014 081.2: The Fall; OT Lesson 4 (Study Notes)
- 01/25/2014 82.1: Cain vs. Enoch; OT Lesson 5 (Core)
- 01/25/2014 82.2: Cain vs. Enoch; OT Lesson 5 (Study Notes)
- 01/31/2014 083.1: The Flood and Tower of Babel; OT Lesson 6 (Core)
- 01/31/2014 083.2: The Flood and Tower of Babel; OT Lesson 6 (Study Notes)
- 02/09/2014 84.1: The Abrahamic Covenant; OT Lesson 7 (Core)
- 02/09/2014 84.2: The Abrahamic Covenant; OT Lesson 7 (Study Notes)
- 02/13/2014 085.1: Living Righteously in a Wicked World; OT Lesson 8 (Core)
- 02/13/2014 085.2: Living Righteously in a Wicked World; OT Lesson 8 (Study Notes)
- 02/20/2014 086:1: The Sacrifice of Isaac; OT Lesson 9 (Core)
- 02/22/2014 086.2: The Sacrifice of Isaac; OT Lesson 9 (Study Notes)
- 03/01/2014 087.1: Birthright and Marriage in the Covenant; OT Lesson 10 (Core)
- 03/02/2014 087.2: Birthright and Marriage in the Covenant; OT Lesson 10 (Study Notes)
- 03/08/2014 088.1: Sexual Morality; OT Lesson 11 (Core)
- 03/08/2014 088.2: Sexual Morality; OT Lesson 11 (Study Notes)
- 03/15/2014 089.1: Joseph in Egypt; OT Lesson 12 (Core)
- 03/18/2014 089.2: Joseph in Egypt; OT Lesson 12 (Study Notes)
- 03/24/2014 090.1: Passover and Exodus; OT Lesson 13 (Core)
- 03/26/2014 090.2: Passover and Exodus; OT Lesson 13 (Study Notes)
- 04/02/2014 091.1: Law and Ritual; OT Lesson 14 (Core)
- 04/13/2014 091.2: Law and Ritual; OT Lesson 14 (Study Notes)
- 04/17/2014 092.1: Look to God and Live; OT Lesson 15 (Core)
- 04/19/2014 092.2: Look to God and Live; OT Lesson 15 (Study Notes)
- 04/20/2014 093.1: I Cannot Go beyond the Word of the Lord; OT Lesson 16 (Core)
- 04/20/2014 093.2: I Cannot Go beyond the Word of the Lord; OT Lesson 16 (Study Notes)
- 05/03/2014 094.1: Farewell of Moses; OT Lesson 17 (Core)
- 05/03/2014 094.2: Farewell of Moses; OT Lesson 17 (Study Notes)
- 05/11/2014 095.1: Be Strong and of Good Courage; OT Lesson 18 (Core)
- 05/11/2014 095.2: Be Strong and of Good Courage; OT Lesson 18 (Study Notes)
- 05/14/2014 096.1: Judges; OT Lesson 19 (Core)
- 05/14/2014 096.2: Judges; OT Lesson 19 (Study Notes)
- 05/22/2014 097.1: Ruth, Woman of Valor; OT Lesson 20 (Core)
- 05/22/2014 097.2: Ruth, Woman of Valor; OT Lesson 20 (Study Notes)
- 05/27/2014 098.1: Honoring God; OT Lesson 21 (Core)
- 05/27/2014 098.2: Honoring God; OT Lesson 21 (Study Notes)
- 06/10/2014 099.1: The Rise of David; OT Lesson 22 (Core)
- 06/10/2014 099.2: The Rise of David; OT Lesson 22 (Study Notes)
- 06/14/2014 100:1: Friendship; OT Lesson 23 (Core)
- 06/14/2014 100:2: Friendship; OT Lesson 23 (Study Notes)
- 06/17/2014 101.1: David and Bathsheba; OT Lesson 24 (Core)
- 06/17/2014 101.2: David and Bathsheba; OT Lesson 24 (Study Notes)
- 07/02/2014 102.1: Psalms and Gratitude; OT Lesson 25 (Core)
- 07/03/2014 102.2: Psalms and Gratitude; OT Lesson 25 (Study Notes)
- 07/13/2014 103.1: The Wisdom and Foolishness of Solomon; OT Lesson 26 (Core)
- 07/17/2014 103.2: The Wisdom and Foolishness of Solomon; OT Lesson 26 (Study Notes)
- 07/18/2014 104.1: Wicked and Righteous Leaders; OT Lesson 27 (Core)
- 07/18/2014 104.2: Wicked and Righteous Leaders; OT Lesson 27 (Study Notes)
- 07/25/2014 105.1: Elijah and the Still Small Voice; OT Lesson 28 (Core)
- 109.1 - 113.2
- 08/14/2014 109.1: Job; OT Lesson 32 (Core)
- 08/15/2014 109.2: Job; OT Lesson 32 (Study Notes)
- 08/23/2014 110.1: Jonah and Micah; OT Lesson 33 (Core)
- 08/24/2014 110.2: Jonah and Micah; OT Lesson 33 (Study Notes)
- 09/01/2014 111.1: Hosea: Fidelity to God and Each Other; OT Lesson 34 (Core)
- 09/01/2014 111.2: Hosea: Fidelity to God and Each Other; OT Lesson 34 (Study Notes)
- 09/05/2014 112.1: Prophets and Social Justice. Amos and Joel; OT Lesson 35 (Core)
- 09/05/2014 112.2: Prophets and Social Justice. Amos and Joel; OT Lesson 35 (Study Notes)
- 09/15/2014 113.1: Introduction to Isaiah; OT Lesson 36 (Core)
- 09/16/2014 113.2: Introduction to Isaiah; OT Lesson 36 (Study Notes)
- 10/01/2014 115.2: Beside Me There Is No Savior; OT Lesson 38 (Study Notes)
- 10/10/2014 116.2: Redemptive Suffering; OT Lesson 39 (Study Notes)
- 11/07/2014 118.2: Jeremiah; OT Lesson 41 (Study Notes)
- 11/08/2014 119.1: Jeremiah and New Covenant; OT Lesson 42 (Core)
- 11/15/2014 120.1: Ezekiel and Shepherds; OT Lesson 43 (Core)
- 11/21/2016 221: Remembered and Nourished (BoM Gospel Doctrine Lesson 47) - doesn't have an audio player embedded in webpage either