Initial Anaconda API

.github/ISSUE_TEMPLATE.md

@@ -0,0 +1,18 @@
+Thanks for taking the time to raise an issue. This template should guide you through the process of submitting a bug, enhancement or feature request. Please erase any part of this template that is not relevant to your issue. 
+
+## Bugs
+Before submitting a bug report:
+
+- [ ] Double-check that the bug is persistent,
+- [ ] Double-check the bug hasn't already been reported [on our issue tracker](https://github.com/issues?utf8=%E2%9C%93&q=is%3Aopen+is%3Aissue+org%3Alibrariesio), they *should* be labelled `bug` or `bugsnag`.
+
+If you have completed those steps then please replace this section with a description of the steps taken to recreate the bug, the expected behavior and the observed behavior.
+
+## Enhancements and Features
+
+Before submitting an enhancement or feature request:
+
+- [ ] Check that the enhancement is not already [in our issue tracker](https://github.com/issues?utf8=%E2%9C%93&q=is%3Aopen+is%3Aissue+org%3Alibrariesio), they should be labelled 'enhancement'., 
+- [ ] For large feature requests, check that your request aligns with our strategy http://docs.libraries.io/strategy.
+
+If you have complete the above step then please replace this section with a description of your proposed enhancement or feature, the motivation for it, an approach and any alternative approaches considered, and whether you are willing and able to create a pull request for it. Note that we may close this issue if it's not something we're planning on working on.

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,10 @@
+Thanks taking the time to contribute. This template should help guide you through the process of creating a pull request for review. Please erase any part of this template that is not relevant to your pull request:
+
+
+- [ ] Have you followed the guidelines for [contributors](http://docs.libraries.io/contributorshandbook)?
+- [ ] Have you checked to ensure there aren't other open pull requests on the repository for a similar change?
+- [ ] Is there a corresponding ticket for your pull request?
+- [ ] Have you written new tests for your changes?
+- [ ] Have you successfully run the project with your changes locally?
+
+If so then please replace this section with a link to the ticket(s) it addressed, an explanation of your change and why you think we should include it. Thanks again! 

.github/SUPPORT.md

@@ -0,0 +1,10 @@
+# Libraries.io Support
+
+If you're looking for support for Libraries.io there are a lot of options, check out:
+
+* Documentation — https://docs.libraries.io
+* Email — support@libraries.io
+* Twitter — https://twitter.com/librariesio
+* Chat — https://slack.libraries.io
+
+On Discuss and in the Libraries.io Slack team, there are a bunch of helpful community members that should be willing to point you in the right direction.

.gitignore

@@ -0,0 +1,3 @@
+log
+coverage
+tmp

.rspec

@@ -0,0 +1 @@
+--require spec_helper

CODE_OF_CONDUCT.md

@@ -0,0 +1,74 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, gender identity and expression, level of experience,
+nationality, personal appearance, race, religion, or sexual identity and
+orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+  address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+  professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at andrew@libraries.io. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at [http://contributor-covenant.org/version/1/4][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/4/

Dockerfile

@@ -0,0 +1,17 @@
+FROM ruby:2.5.0-alpine
+RUN apk add --update \
+  build-base git curl-dev \
+  && rm -rf /var/cache/apk/*
+
+# throw errors if Gemfile has been modified since Gemfile.lock
+RUN bundle config --global frozen 1
+RUN mkdir -p /usr/src/app
+WORKDIR /usr/src/app
+
+ENV RACK_ENV production
+
+COPY Gemfile Gemfile.lock /usr/src/app/
+RUN bundle install --without test --jobs 2
+
+COPY . /usr/src/app
+CMD puma

Gemfile

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+source "https://rubygems.org"
+ruby "2.5.0"
+
+gem "builder"
+gem "feedtosis", git: "https://github.com/alown/feedtosis"
+gem "foreman"
+gem "hiredis"
+gem "httparty"
+gem "moneta"
+gem "msgpack"
+gem "puma"
+gem "redis"
+gem "sinatra"
+
+group :test do
+  gem "fakeredis", require: "fakeredis/rspec"
+  gem "pry"
+  gem "rack-test"
+  gem "rspec"
+end

Gemfile.lock

@@ -0,0 +1,94 @@
+GIT
+  remote: https://github.com/alown/feedtosis
+  revision: d763f2cdae9d75a42f9b04b03bc28b71b57c4525
+  specs:
+    feedtosis (0.0.3.6)
+      curb
+      feed-normalizer
+      http_headers
+      moneta
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    builder (3.2.3)
+    coderay (1.1.2)
+    curb (0.9.6)
+    diff-lcs (1.3)
+    fakeredis (0.7.0)
+      redis (>= 3.2, < 5.0)
+    feed-normalizer (1.5.2)
+      hpricot (>= 0.6)
+      simple-rss (>= 1.1)
+    foreman (0.85.0)
+      thor (~> 0.19.1)
+    hiredis (0.6.1)
+    hpricot (0.8.6)
+    http_headers (0.0.2.3)
+    httparty (0.17.0)
+      mime-types (~> 3.0)
+      multi_xml (>= 0.5.2)
+    method_source (0.9.2)
+    mime-types (3.3)
+      mime-types-data (~> 3.2015)
+    mime-types-data (3.2019.0904)
+    moneta (1.0.0)
+    msgpack (1.2.4)
+    multi_xml (0.6.0)
+    mustermann (1.0.2)
+    pry (0.12.2)
+      coderay (~> 1.1.0)
+      method_source (~> 0.9.0)
+    puma (3.11.4)
+    rack (2.0.5)
+    rack-protection (2.0.3)
+      rack
+    rack-test (1.1.0)
+      rack (>= 1.0, < 3)
+    redis (4.0.1)
+    rspec (3.8.0)
+      rspec-core (~> 3.8.0)
+      rspec-expectations (~> 3.8.0)
+      rspec-mocks (~> 3.8.0)
+    rspec-core (3.8.2)
+      rspec-support (~> 3.8.0)
+    rspec-expectations (3.8.4)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.8.0)
+    rspec-mocks (3.8.1)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.8.0)
+    rspec-support (3.8.2)
+    simple-rss (1.3.3)
+    sinatra (2.0.3)
+      mustermann (~> 1.0)
+      rack (~> 2.0)
+      rack-protection (= 2.0.3)
+      tilt (~> 2.0)
+    thor (0.19.4)
+    tilt (2.0.8)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  builder
+  fakeredis
+  feedtosis!
+  foreman
+  hiredis
+  httparty
+  moneta
+  msgpack
+  pry
+  puma
+  rack-test
+  redis
+  rspec
+  sinatra
+
+RUBY VERSION
+   ruby 2.5.0p0
+
+BUNDLED WITH
+   1.17.3

LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2019 Copyright Tidelift, Inc.
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Procfile

@@ -0,0 +1 @@
+web: puma config.ru

Readme.md

@@ -0,0 +1,15 @@
+# Conda API
+
+A small service to make it easier for [Libraries.io](https://libraries.io) to read data about Conda Packages in different channels.
+
+## Essentials
+
+- Provide a REST interface for list of all names of packages (as json)
+- Provide a REST interface for list of versions for each package (as json)
+- Update info from Specs repo frequently
+
+## Extras
+
+- Watch https://github.com/Conda/Specs/commits/master.atom for updates
+- RSS feed of new/updated packages for https://github.com/librariesio/dispatcher to track
+- Tell Libraries about removed versions/packages

app.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+require "sinatra/base"
+require "./conda_repo"
+require "builder"
+
+redis = Redis.new(url: ENV["REDIS_SERVER"], driver: :hiredis)
+CONDA_REPO = CondaRepo.new(redis)
+CONDA_REPO.update_packages
+
+class CondaAPI < Sinatra::Base
+  get "/" do
+    "Hello World! #{CONDA_REPO.package_names.length} \n"
+  end
+
+  get "/packages.json" do
+    content_type :json
+    CONDA_REPO.package_names.to_json
+  end
+
+  get "/packages/:name.json" do
+    content_type :json
+    channel = "pkgs/main" # Default for now
+    package = CONDA_REPO.package(channel, params[:name])
+    if package
+      package.to_json
+    else
+      halt 404, "Product not found"
+    end
+  end
+end

conda_repo.rb

@@ -0,0 +1,63 @@
+# frozen_string_literal: true
+
+require "msgpack"
+require "json"
+require "httparty"
+require "net/http"
+
+class CondaRepo
+  CHANNEL_PATH = File.dirname(__FILE__) + "/tmp/channels"
+  CHANNELS = [
+    # channel, domain, directory_path_to channeldata.json
+    ["pkgs/main", "repo.anaconda.com"],
+    # ["conda-forge", "conda.anaconda.org", "/conda-forge"],
+  ].freeze
+
+  def initialize(redis)
+    @redis = redis
+  end
+
+  def package_names
+    @redis.smembers("package_names")
+  end
+
+  def package(channel, name)
+    pack = @redis.get("packages:#{channel}/#{name}")
+    return nil unless pack
+
+    MessagePack.unpack(pack.force_encoding("ASCII-8BIT"))
+  end
+
+  def download_and_parse_packages
+    channels_with_packages = {}
+    CHANNELS.each do |channel, domain|
+      channel_packages = download_json(channel, domain)["packages"]
+      channels_with_packages[channel] ||= {}
+      channel_packages.each do |package_name, package|
+        channels_with_packages[channel][package_name] = package
+      end
+    end
+
+    channels_with_packages
+  end
+
+  def update_packages
+    download_and_parse_packages.each do |channel, packages|
+      @redis.sadd("package_names", packages.keys.map { |name| "#{channel}/#{name}" })
+
+      packages.each_with_index do |(name, package_info), index|
+        version = package_info["version"]
+        key = "#{channel}/#{name}"
+        @redis.set("packages:#{key}", MessagePack.pack(version))
+        GC.start if index % 100 == 0
+      end
+    end
+  end
+
+  private
+
+  def download_json(channel, domain)
+    url = "https://#{domain}/#{channel}/channeldata.json"
+    HTTParty.get(url).parsed_response
+  end
+end